Hi. I have a Shuttle box with an AMD Athlon XP 2200+ and 1GB of RAM. I''m normally running it with Debian sarge/sid and kernel 2.6.10-1-k7, as built by Debian. I want to use Xen on it. I built a xen0 kernel which is as close to the Debian kernel as I can (no power management, no HPET timers, broken ISA drivers disabled), disabled /lib/tls, and booted with the new kernel. Everything works. I can run domUs as I want. But, the performance is horrible, even with just dom0 running. I tested the disk, it still transfers ~50MB/s. I don''t seem to be suffering from PIO mode. Also, things running from RAM cache are as slow as from disk. I ran some of the lmbench tests both under xen and with native linux. Here are the results, with the "-" lines from xen and "+" lines from native linux. The results are pretty poor, and so is the feel of the machine under xen. Any hints on what to try next? bw_mem -0.131072 1068.95 +0.131072 4409.00 bw_pipe -Pipe bandwidth: 85.42 MB/sec +Pipe bandwidth: 989.19 MB/sec bw_unix -AF_UNIX sock stream bandwidth: 104.75 MB/sec +AF_UNIX sock stream bandwidth: 968.87 MB/sec lat_ctx -"size=0k ovr=3.15 -10 13.95 +"size=0k ovr=1.09 +10 0.94 lat_fcntl -Fcntl lock latency: 43.2246 microseconds +Fcntl lock latency: 5.4676 microseconds lat_fifo -FIFO latency: 32.4076 microseconds +FIFO latency: 3.3622 microseconds lat_fs -0k 1000 18563 31549 -1k 1000 6426 17887 -4k 1000 6583 17826 -10k 1000 3084 14989 +0k 1000 63666 92541 +1k 1000 24125 51193 +4k 1000 25863 52089 +10k 1000 12533 41564 lat_pipe -Pipe latency: 31.0475 microseconds +Pipe latency: 5.4833 microseconds lat_proc -Process fork+exit: 789.2857 microseconds +Process fork+exit: 113.7347 microseconds lat_select -Select on 200 tcp fd''s: 46.8159 microseconds +Select on 200 tcp fd''s: 12.9369 microseconds lat_syscall -Simple syscall: 0.6188 microseconds +Simple syscall: 0.1547 microseconds lat_unix -AF_UNIX sock stream latency: 61.1547 microseconds +AF_UNIX sock stream latency: 21.7704 microseconds ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Tommi Virtanen wrote:> I tested the disk, it still transfers ~50MB/s. I don''t seem to be > suffering from PIO mode. Also, things running from RAM cache are as > slow as from disk.What version of Xen is this?> Any hints on what to try next?If you''re running xen-unstable, try xen-2.0.x or vice versa. Are you absolutely positive that you''ve got enough memory allocated to dom0? That you''ve got the proper IO chipset support built into the kernel? That you''ve really got tls disabled (if you just move tls after an app has already launched using it it will still be using it until you restart the app)? These numbers seem too bad to be Xen''s fault alone. Do you have swap enabled? Is there a great deal of swap activity? Regards, Anthony Liguori> bw_mem > -0.131072 1068.95 > +0.131072 4409.00 > > bw_pipe > -Pipe bandwidth: 85.42 MB/sec > +Pipe bandwidth: 989.19 MB/sec > > bw_unix > -AF_UNIX sock stream bandwidth: 104.75 MB/sec > +AF_UNIX sock stream bandwidth: 968.87 MB/sec > > lat_ctx > -"size=0k ovr=3.15 > -10 13.95 > +"size=0k ovr=1.09 > +10 0.94 > > lat_fcntl > -Fcntl lock latency: 43.2246 microseconds > +Fcntl lock latency: 5.4676 microseconds > > lat_fifo > -FIFO latency: 32.4076 microseconds > +FIFO latency: 3.3622 microseconds > > lat_fs > -0k 1000 18563 31549 > -1k 1000 6426 17887 > -4k 1000 6583 17826 > -10k 1000 3084 14989 > +0k 1000 63666 92541 > +1k 1000 24125 51193 > +4k 1000 25863 52089 > +10k 1000 12533 41564 > > lat_pipe > -Pipe latency: 31.0475 microseconds > +Pipe latency: 5.4833 microseconds > > lat_proc > -Process fork+exit: 789.2857 microseconds > +Process fork+exit: 113.7347 microseconds > > lat_select > -Select on 200 tcp fd''s: 46.8159 microseconds > +Select on 200 tcp fd''s: 12.9369 microseconds > > lat_syscall > -Simple syscall: 0.6188 microseconds > +Simple syscall: 0.1547 microseconds > > lat_unix > -AF_UNIX sock stream latency: 61.1547 microseconds > +AF_UNIX sock stream latency: 21.7704 microseconds > > > ------------------------------------------------------- > SF email is sponsored by - The IT Product Guide > Read honest & candid reviews on hundreds of IT Products from real users. > Discover which products truly live up to the hype. Start reading now. > http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/xen-devel >------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> Hi. I have a Shuttle box with an AMD Athlon XP 2200+ and 1GB of RAM. > I''m normally running it with Debian sarge/sid and kernel 2.6.10-1-k7, > as built by Debian. I want to use Xen on it. I built a xen0 kernel > which is as close to the Debian kernel as I can (no power management, > no HPET timers, broken ISA drivers disabled), disabled /lib/tls, and > booted with the new kernel.> lat_syscall > -Simple syscall: 0.6188 microseconds > +Simple syscall: 0.1547 microsecondsSystem call latency should be identical between Xen and native. The fact that it is not inidcates that your system is seriously screwed running Xen. Have you got some interrupt line going crazy? Ian ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_ide95&alloc_id396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> Hi. I have a Shuttle box with an AMD Athlon XP 2200+ and 1GB of RAM. > I''m normally running it with Debian sarge/sid and kernel 2.6.10-1-k7, > as built by Debian. I want to use Xen on it. I built a xen0 kernel > which is as close to the Debian kernel as I can (no power management, > no HPET timers, broken ISA drivers disabled), disabled /lib/tls, and > booted with the new kernel.Please try using our kernel config. BTW: you''re not doing anything rash like using an SMP enabled kernel are you? There''s still loads of debugging turned on at the moment. Ian> Everything works. I can run domUs as I want. But, the performance is > horrible, even with just dom0 running. > > I tested the disk, it still transfers ~50MB/s. I don''t seem to be > suffering from PIO mode. Also, things running from RAM cache are as > slow as from disk. > > I ran some of the lmbench tests both under xen and with native linux. > Here are the results, with the "-" lines from xen and "+" lines from > native linux. The results are pretty poor, and so is the feel of the > machine under xen. > > Any hints on what to try next? > > > bw_mem > -0.131072 1068.95 > +0.131072 4409.00 > > bw_pipe > -Pipe bandwidth: 85.42 MB/sec > +Pipe bandwidth: 989.19 MB/sec > > bw_unix > -AF_UNIX sock stream bandwidth: 104.75 MB/sec > +AF_UNIX sock stream bandwidth: 968.87 MB/sec > > lat_ctx > -"size=0k ovr=3.15 > -10 13.95 > +"size=0k ovr=1.09 > +10 0.94 > > lat_fcntl > -Fcntl lock latency: 43.2246 microseconds > +Fcntl lock latency: 5.4676 microseconds > > lat_fifo > -FIFO latency: 32.4076 microseconds > +FIFO latency: 3.3622 microseconds > > lat_fs > -0k 1000 18563 31549 > -1k 1000 6426 17887 > -4k 1000 6583 17826 > -10k 1000 3084 14989 > +0k 1000 63666 92541 > +1k 1000 24125 51193 > +4k 1000 25863 52089 > +10k 1000 12533 41564 > > lat_pipe > -Pipe latency: 31.0475 microseconds > +Pipe latency: 5.4833 microseconds > > lat_proc > -Process fork+exit: 789.2857 microseconds > +Process fork+exit: 113.7347 microseconds > > lat_select > -Select on 200 tcp fd''s: 46.8159 microseconds > +Select on 200 tcp fd''s: 12.9369 microseconds > > lat_syscall > -Simple syscall: 0.6188 microseconds > +Simple syscall: 0.1547 microseconds > > lat_unix > -AF_UNIX sock stream latency: 61.1547 microseconds > +AF_UNIX sock stream latency: 21.7704 microseconds > > > ------------------------------------------------------- > SF email is sponsored by - The IT Product Guide > Read honest & candid reviews on hundreds of IT Products from > real users. > Discover which products truly live up to the hype. Start reading now. > http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/xen-devel >------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_ide95&alloc_id396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Anthony Liguori wrote:>> I tested the disk, it still transfers ~50MB/s. I don''t seem to be >> suffering from PIO mode. Also, things running from RAM cache are as >> slow as from disk. > What version of Xen is this?2.0.4, as packaged for Debian by Adam Heath at http://people.debian.org/~doogie/> If you''re running xen-unstable, try xen-2.0.x or vice versa. Are you > absolutely positive that you''ve got enough memory allocated to dom0?As dom0 is my main environment, it gets almost all of the RAM. title Xen 2.6.10 root (hd0,1) kernel /boot/xen.gz dom0_mem=917504 module /boot/xen-linux-2.6.10+xen0 root=/dev/hda2 ro module /boot/initrd.img-2.6.10+xen0 savedefault boot> That you''ve got the proper IO chipset support built into the kernel?The config is pretty much identical between native linux and xen dom0.> That you''ve really got tls disabled (if you just move tls after an app > has already launched using it it will still be using it until you > restart the app)?I have rebooted many times since the move.> These numbers seem too bad to be Xen''s fault alone. Do you have swap > enabled? Is there a great deal of swap activity?There''s 2 gigs of swap with under 3 megabytes used, no blocks going in or out. ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Ian Pratt wrote:>>as built by Debian. I want to use Xen on it. I built a xen0 kernel >>which is as close to the Debian kernel as I can (no power management, > Please try using our kernel config.I did that first, the interactive feel was as poor, and that''s the reason I went for the config that was as close to my normal native config as possible, to try to eliminate driver issues.> BTW: you''re not doing anything rash like using an SMP enabled kernel are > you? There''s still loads of debugging turned on at the moment.# CONFIG_SMP is not set Will get back to you with interrupt statistics as soon as I have time to reboot this machine. ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Ian Pratt wrote:> Have you got some interrupt line going crazy?That seems to be the problem. In native linux, I see: 0 timer 1000/s LOC 1000/s and in xen dom0 I see: 21 uhci_hcd 35000/s 130 timer 45000/s rmmodding uhci_hcd removes irq 21, but makes the timer speed up to 115000/s. rmmod ohci13943, rmmod ehci_hcd makes the timer go 800000/s. rmmoding even more drivers puts the peak at 1100000/s, until it suddenly drops to 30/s. Didn''t check between each rmmod, so I don''t know which exact one caused it. Unplugging my USB trackball and modprobing _all_ of the drivers above causes timer to tick at 60/s. Plugging in my USB trackball makes it 33000/s. Unplugging has no effect. Booting up with trackball unconnected gives about 30000/s, too, so it''s not just the USB mouse driver. It seems there is a problem with USB. But that doesn''t explain everything. It seems the bridge module has trouble, too. rmmod bridge, or just ip li set dev br0 down makes timer go from crazy to normal, too. Thanks for the hint, I think I can manage to find a working configuration based on this. Of course, having no such problems in the first place would be even better. ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Ian Pratt wrote:> Just to be clear, you''re not using the usb virtualization stuff, just > accessing the device in domain 0?During these tests, no domU was ever started.> Does anyone else see this interrupt storm with USB devices active? > Please can everyone check as I''ll hold the 2.0.5 release (with USB > enabled in the default config) until we understand the scale of the > problem.Also, note that I can get the same problem with bridging: lots of timer interrupts ip li set dev br0 down --> timer gets 30/s ip li set dev br0 up --> timer goes crazy ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> Ian Pratt wrote: > > Have you got some interrupt line going crazy? > > That seems to be the problem. > > In native linux, I see: > > 0 timer 1000/s > LOC 1000/s > > and in xen dom0 I see: > > 21 uhci_hcd 35000/s > 130 timer 45000/s > > rmmodding uhci_hcd removes irq 21, but makes the timer > speed up to 115000/s. > > rmmod ohci13943, rmmod ehci_hcd makes the timer go > 800000/s.Just to be clear, you''re not using the usb virtualization stuff, just accessing the device in domain 0? Does anyone else see this interrupt storm with USB devices active? Please can everyone check as I''ll hold the 2.0.5 release (with USB enabled in the default config) until we understand the scale of the problem. Thanks, Ian ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_ide95&alloc_id396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> Also, note that I can get the same problem with bridging: > > lots of timer interrupts > ip li set dev br0 down --> timer gets 30/s > ip li set dev br0 up --> timer goes crazyI certainly don''t see this on any of my systems. Are you using the standard bridging scripts? I guess not since your bridge name is different. The only time I''ve seen something like this is when some of the bridge timeout parameters are set to 0 jiffies (setting to 1 jiffy is OK). Ian ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_ide95&alloc_id396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Ian Pratt wrote:> Are you using the standard bridging scripts? I guess not since your > bridge name is different. The only time I''ve seen something like this is > when some of the bridge timeout parameters are set to 0 jiffies (setting > to 1 jiffy is OK).I''m not. I had a fully working UML setup and wanted to share that between UML and Xen. The 0 jiffie thing was a good hint. That was the problem with bridge. I''m guessing I had that issue all along with my native linux setup. Summary: Problem #1: plugging in my Logitech USB trackball creates an uhci_hcd and timer interrupt storm that degrades performance to about 10% of normal. Unplug does not stop storm. Unknown cause, not fixed, workaround is not to plug in any USB device except hubs. Problem #2: bridge setfd/sethello 0 causes timer interrupt storm, but does not degrade performance significantly. Workaround is to not set such low numbers. Will talk to brctl/bridging maintainers to make it more foolproof. ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> Problem #1: plugging in my Logitech USB trackball creates an uhci_hcd > and timer interrupt storm that degrades performance to about 10% of > normal. Unplug does not stop storm. Unknown cause, not fixed, > workaround > is not to plug in any USB device except hubs.Could you try using some of the kernel profile stuff to figure out where the time is being spent?> Problem #2: bridge setfd/sethello 0 causes timer interrupt storm, but > does not degrade performance significantly. Workaround is to not set > such low numbers. Will talk to brctl/bridging maintainers to make it > more foolproof.If you''re starting a dialogue with the bridge folk, I''d be grateful if you could bring up the issue of transferring IP addresses from interfaces that are added to a bridge to the bridge. Thanks, Ian ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_ide95&alloc_id396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Ian Pratt wrote:> If you''re starting a dialogue with the bridge folk, I''d be grateful > if you could bring up the issue of transferring IP addresses from > interfaces that are added to a bridge to the bridge.Transferring IP addresses is not enough - if there is a dhclient running for eth0 on the host, there is no way to change it to be running for br0 transparently. So all we get is a seemingly well working system - until the dhclient renews the lease. The only solution to these kind of issues globally that I''d see, would be the possibility to have packets come "up" from the eth0 interface, instead of the br0 interface - which would need to be somehow configurable. -- Naked ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Tommi Virtanen wrote:> Problem #2: bridge setfd/sethello 0 causes timer interrupt storm, > but does not degrade performance significantly. Workaround is to not > set such low numbers. Will talk to brctl/bridging maintainers to > make it more foolproof.This is really odd. We run a lot of bridges with such settings. Or, to be specific, for us, ''setfd'' is always 0, ''setageing'' is either 0 or 300, depending if we want the bridge to act as a ''hub'' or not. STP is always off. Under these circumstances, we have never seen a timer interrupt storm. But perhaps it is particular to having STP on. -- Naked ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Can you do a "lspci -v" as root and see if there is any correlation between your network adapters and your usb devices? (same interrupt) or something. Thanks, B. On Mon, 2005-03-07 at 15:19, Nuutti Kotivuori wrote:> Tommi Virtanen wrote: > > Problem #2: bridge setfd/sethello 0 causes timer interrupt storm, > > but does not degrade performance significantly. Workaround is to not > > set such low numbers. Will talk to brctl/bridging maintainers to > > make it more foolproof. > > This is really odd. We run a lot of bridges with such settings. Or, to > be specific, for us, ''setfd'' is always 0, ''setageing'' is either 0 or > 300, depending if we want the bridge to act as a ''hub'' or not. STP is > always off. > > Under these circumstances, we have never seen a timer interrupt > storm. But perhaps it is particular to having STP on. > > -- Naked > > > > > ------------------------------------------------------- > SF email is sponsored by - The IT Product Guide > Read honest & candid reviews on hundreds of IT Products from real users. > Discover which products truly live up to the hype. Start reading now. > http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/xen-devel >------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel