Using an 8-way ES7000 with 32GB of ram, I''m running xen at changset 11217 with dom0_mem=3092M as a kernel parameter on a x86_64 hyperviser. It appears that qemu is adding too much overhead to virtual machines to display graphics. If I boot a sles9sp3 domVT in runlevel 5 with vga=0x314 for a graphicalboot, and immediately use vncviewer simply to watch the virtual machine as it boots, qemu-dm consumes 40% cpu as soon as vncviewer is launched and remains that way throughout this entire process without settling down when the login screen appears. Even If I close the console that''s viewing the virtual host, qemu-dm remains at 40-44%. When booting the same domVT in runlevel 5 without ever attempting to connect to it with vncviewer, qemu-dm fluctuated from 1-3% while the host was booting and then settled to 0.2% after 4-5 mins. Assuming that the boot process was complete, I launched vncviewer and qemu-dm jumped up to a 40-44% range again. Lastly I started the same host in runlevel 3 in pure text mode without any vga parameters on the kernel line, connected to the virtual machine with vncviewer, and watched the machine boot until a login prompt appeared. At this point, top showed qemu-dm as using 0% of the cpu. Logging into the virtual machine and running ''startx'' however sent qemu-dm back up to a 40- 44% range. Why is qemu-dm consuming so much of the cpu? Why is qemu-dm still high even when I close my vncviewer? Tommie, Xen Test Team Unisys _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> It appears that qemu is adding too much overhead to > virtual machines to display graphics. > If I boot a sles9sp3 domVT in runlevel 5 with vga=0x314 for a > graphicalboot, and immediately use > vncviewer simply to watch the virtual machine as it boots, > qemu-dm consumes 40% cpu as soon as vncviewer is launched and remainsthat> way throughout this entire process without settling down when thelogin> screen appears. > Even If I close the console that''s viewing the virtual host, qemu-dm > remains at 40-44%.That''s considerably more than my experience. With a vncviewer connected it''s normal to see qemu-dm burning 5-10% CPU. We do the majority of our HVM testing using windows as obviously the best way of running linux is fully ''enlightened'' (para-virtualized). It would be good to test to see whether the issue that you''re seeing is sles9 specific. Thanks, Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian,> We do the majority of our HVM testing using windows as obviously the > best way of running linux is fully ''enlightened'' (para-virtualized).It> would be good to test to see whether the issue that you''re seeing is > sles9 specific.We''ve actually been doing most of our HVM testing with WinXP. Using WinXP we found that if we never ran VNC -- waited long enough that we were sure the WinXP guests were running, and then accessed them via remote desktop -- we did not see the high cpu utilization. But, as soon as we fired-up VNC, we encountered the high utilization that Tommie described. Tommie posted the Linux results because most of us have better tools to investigate Linux, as opposed to Windows, problems. Are you seeing similar issues with Windows? Cheers, Aravindh Puthiyaparambil Xen Development Team Unisys, Tredyffrin PA _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > We do the majority of our HVM testing using windows as obviously the > > best way of running linux is fully ''enlightened'' (para-virtualized). > It > > would be good to test to see whether the issue that you''re seeing is > > sles9 specific. > > We''ve actually been doing most of our HVM testing with WinXP. Using > WinXP we found that if we never ran VNC -- waited long enough that we > were sure the WinXP guests were running, and then accessed them via > remote desktop -- we did not see the high cpu utilization. But, assoon> as we fired-up VNC, we encountered the high utilization that Tommie > described. Tommie posted the Linux results because most of us have > better tools to investigate Linux, as opposed to Windows, problems. > > Are you seeing similar issues with Windows?No, at least with w2k3. Could you try and repro with w2k3? What VNC client are you using? Thanks, Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> No, at least with w2k3. Could you try and repro with w2k3?Will do. BTW, we are seeing the same issue if we use SDL.> What VNC client are you using?TightVNC viewer version 1.2.9> Thanks, > Ian_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> No, at least with w2k3. Could you try and repro with w2k3?I''m getting the same results with win2k3 that I get with the winXP and sles9, so it''s probably not a sles9 issue. I started the win2k3 guest and waited about 4-5 mins for it to finish booting. At this point qemu-dm used 0% CPU. When I launched vncviewer to display the win2k3 guest, qemu-dm jumped to 40%-45%. What do you think is causing this to happen? Tommie McAfee Xen-Testing -----Original Message----- From: Puthiyaparambil, Aravindh Sent: Mon 8/28/2006 11:00 AM To: Ian Pratt; McAfee, Tommie M; xen-devel@lists.xensource.com Cc: ian.pratt@cl.cam.ac.uk Subject: RE: [Xen-devel] qemu-dm cpu utilization> No, at least with w2k3. Could you try and repro with w2k3?Will do. BTW, we are seeing the same issue if we use SDL.> What VNC client are you using?TightVNC viewer version 1.2.9> Thanks, > Ian_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Another pertinent question is, what size screen are you using?>From what I''ve heard, rather than trap on every MMIO instruction, HVMguests are given direct access to the virtual framebuffer. But this means that QEMU needs to scan the framebuffer on a regular basis to find changes, and then do diffs to make appropriate VNC changes. This probably scales linearly with the number of pixels to be scanned. Try using a lower resolution, and see what you get. -George On 8/28/06, McAfee, Tommie M <Tommie.McAfee@unisys.com> wrote:> > > No, at least with w2k3. Could you try and repro with w2k3? > > I''m getting the same results with win2k3 that I get with the winXP and sles9, so it''s probably not a sles9 issue. I started the win2k3 guest and waited about 4-5 mins for it to finish booting. At this point qemu-dm used 0% CPU. When I launched vncviewer to display the win2k3 guest, qemu-dm jumped to 40%-45%. > > What do you think is causing this to happen? > > Tommie McAfee > Xen-Testing > > > > -----Original Message----- > From: Puthiyaparambil, Aravindh > Sent: Mon 8/28/2006 11:00 AM > To: Ian Pratt; McAfee, Tommie M; xen-devel@lists.xensource.com > Cc: ian.pratt@cl.cam.ac.uk > Subject: RE: [Xen-devel] qemu-dm cpu utilization > > > No, at least with w2k3. Could you try and repro with w2k3? > > Will do. BTW, we are seeing the same issue if we use SDL. > > > What VNC client are you using? > > TightVNC viewer version 1.2.9 > > > Thanks, > > Ian > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Another pertinent question is, what size screen are you using? > > From what I''ve heard, rather than trap on every MMIO > instruction, HVM guests are given direct access to the > virtual framebuffer. But this means that QEMU needs to scan > the framebuffer on a regular basis to find changes, and then > do diffs to make appropriate VNC changes. This probably > scales linearly with the number of pixels to be scanned. > > Try using a lower resolution, and see what you get.1024x768 is the ''recommended''. I don''t think the emulated Cirrus card had enough video memory to do full-colour in higher resoloutions. Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > Try using a lower resolution, and see what you get. > > 1024x768 is the ''recommended''. I don''t think the emulated Cirrus card > had enough video memory to do full-colour in higher resoloutions.We have been doing all our testing at 800x600 and I don''t think we can go lower than that with W2k3. Any pointers to why this could be happening? Thanks, Aravindh _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > 1024x768 is the ''recommended''. I don''t think the emulated Cirruscard> > had enough video memory to do full-colour in higher resoloutions. > > We have been doing all our testing at 800x600 and I don''t think we can > go lower than that with W2k3. > > Any pointers to why this could be happening?Are you explicitly setting 800x600? I thought w2k3 installs defaulted to 1024x768, but maybe I''m confused. When you disconnect the vnc console qemu should stop scanning the framebuffer RAM. The fact that the CPU usage doesn''t go down is odd. Perhaps try attaching strace or gdb to the qemu-dm to get an idea of what the loops is? Thanks, Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
McAfee, Tommie M wrote:> Using an 8-way ES7000 with 32GB of ram, I''m running xen at changset 11217 > with dom0_mem=3092M as a kernel parameter on a x86_64 hyperviser. > > It appears that qemu is adding too much overhead to > virtual machines to display graphics. > If I boot a sles9sp3 domVT in runlevel 5 with vga=0x314 for a graphicalboot, and immediately use > vncviewer simply to watch the virtual machine as it boots, > qemu-dm consumes 40% cpu as soon as vncviewer is launched and remains that way throughout this entire process without settling down when the login screen appears. > Even If I close the console that''s viewing the virtual host, qemu-dm remains at 40-44%.Almost everything in the VNC code resets when a guest disconnected except for the timer. If a client disconnects, the timer routine should be almost a nop but it''s possible that it''s somehow causing qemu-dm to do something funky. If you delete the timer (and set vs->timer = NULL) in vnc_client_io_error(), does it reduce the usage? BTW, if you''re using 800x600, I suspect that you may be falling back to MMIO instead of a linear framebuffer. Try moving to 1024x768 and see if that helps. Regards, Anthony Liguori> > When booting the same domVT in runlevel 5 without ever attempting to connect to it with > vncviewer, qemu-dm fluctuated from 1-3% while the host was booting and then settled to 0.2% after 4-5 mins. Assuming that the boot process was complete, I launched vncviewer and qemu-dm jumped up to a 40-44% range again. > > Lastly I started the same host in runlevel 3 in pure text mode without any > vga parameters on the kernel line, connected to the virtual machine with > vncviewer, and watched the machine boot until a login prompt appeared. At > this point, top showed qemu-dm as using 0% of the cpu. Logging into the > virtual machine and running ''startx'' however sent qemu-dm back up to a 40- > 44% range. > > Why is qemu-dm consuming so much of the cpu? Why is qemu-dm still high > even when I close my vncviewer? > > Tommie, > Xen Test Team > Unisys > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Anthony, thanks for the suggestion. I defiantly think that the resolution of 800x600 was causing some kind of problem (perhaps the linear framebuffer). When I set it to 1024x768 qemu-dm went down to 10%-12%. Is this the normal usage that you are seeing? Tommie McAfee Xen-Testing -----Original Message----- From: Anthony Liguori [mailto:anthony@codemonkey.ws] Sent: Tue 8/29/2006 7:43 PM To: McAfee, Tommie M Subject: Re: qemu-dm cpu utilization McAfee, Tommie M wrote:> Using an 8-way ES7000 with 32GB of ram, I''m running xen at changset 11217 > with dom0_mem=3092M as a kernel parameter on a x86_64 hyperviser. > > It appears that qemu is adding too much overhead to > virtual machines to display graphics. > If I boot a sles9sp3 domVT in runlevel 5 with vga=0x314 for a graphicalboot, and immediately use > vncviewer simply to watch the virtual machine as it boots, > qemu-dm consumes 40% cpu as soon as vncviewer is launched and remains that way throughout this entire process without settling down when the login screen appears. > Even If I close the console that''s viewing the virtual host, qemu-dm remains at 40-44%.Almost everything in the VNC code resets when a guest disconnected except for the timer. If a client disconnects, the timer routine should be almost a nop but it''s possible that it''s somehow causing qemu-dm to do something funky. If you delete the timer (and set vs->timer = NULL) in vnc_client_io_error(), does it reduce the usage? BTW, if you''re using 800x600, I suspect that you may be falling back to MMIO instead of a linear framebuffer. Try moving to 1024x768 and see if that helps. Regards, Anthony Liguori> > When booting the same domVT in runlevel 5 without ever attempting to connect to it with > vncviewer, qemu-dm fluctuated from 1-3% while the host was booting and then settled to 0.2% after 4-5 mins. Assuming that the boot process was complete, I launched vncviewer and qemu-dm jumped up to a 40-44% range again. > > Lastly I started the same host in runlevel 3 in pure text mode without any > vga parameters on the kernel line, connected to the virtual machine with > vncviewer, and watched the machine boot until a login prompt appeared. At > this point, top showed qemu-dm as using 0% of the cpu. Logging into the > virtual machine and running ''startx'' however sent qemu-dm back up to a 40- > 44% range. > > Why is qemu-dm consuming so much of the cpu? Why is qemu-dm still high > even when I close my vncviewer? > > Tommie, > Xen Test Team > Unisys > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Anthony,>If you delete the timer (and set vs->timer = NULL) in >vnc_client_io_error(), does it reduce the usage?setting vs->timer=NULL causes qemu-dm to segfault when I launch vncviewer to view the virtual host. Here is a snippet of the strace results: ioctl(13, EVIOCGKEYCODE, 0x7fffff904ee0) = 0 select(17, [6 9 11 13 16], [], [], {0, 10000}) = 1 (in [13], left {0, 10000}) read(13, "\34\0\0\0", 4) = 4 write(13, "\34\0\0\0", 4) = 4 clock_gettime(CLOCK_MONOTONIC, {87584, 18585008}) = 0 clock_gettime(CLOCK_MONOTONIC, {87584, 18684008}) = 0 clock_gettime(CLOCK_MONOTONIC, {87584, 18783008}) = 0 ioctl(13, EVIOCGKEYCODE, 0x7fffff904ee0) = 0 select(17, [6 9 11 13 16], [], [], {0, 10000}) = 1 (in [13], left {0, 10000}) read(13, "\34\0\0\0", 4) = 4 write(13, "\34\0\0\0", 4) = 4 clock_gettime(CLOCK_MONOTONIC, {87584, 31583008}) = 0 clock_gettime(CLOCK_MONOTONIC, {87584, 31685008}) = 0 clock_gettime(CLOCK_MONOTONIC, {87584, 31785008}) = 0 clock_gettime(CLOCK_MONOTONIC, {87584, 57104008}) = 0 --- SIGSEGV (Segmentation fault) @ 0 (0) --- Process 17185 detached According to strace, the timers account for a lot of qemu-dm''s activity. Is this polling a bit excessive? Tommie McAfee Xen-Testing -----Original Message----- From: Anthony Liguori [mailto:anthony@codemonkey.ws] Sent: Tue 8/29/2006 7:43 PM To: McAfee, Tommie M Subject: Re: qemu-dm cpu utilization McAfee, Tommie M wrote:> Using an 8-way ES7000 with 32GB of ram, I''m running xen at changset 11217 > with dom0_mem=3092M as a kernel parameter on a x86_64 hyperviser. > > It appears that qemu is adding too much overhead to > virtual machines to display graphics. > If I boot a sles9sp3 domVT in runlevel 5 with vga=0x314 for a graphicalboot, and immediately use > vncviewer simply to watch the virtual machine as it boots, > qemu-dm consumes 40% cpu as soon as vncviewer is launched and remains that way throughout this entire process without settling down when the login screen appears. > Even If I close the console that''s viewing the virtual host, qemu-dm remains at 40-44%.Almost everything in the VNC code resets when a guest disconnected except for the timer. If a client disconnects, the timer routine should be almost a nop but it''s possible that it''s somehow causing qemu-dm to do something funky. If you delete the timer (and set vs->timer = NULL) in vnc_client_io_error(), does it reduce the usage? BTW, if you''re using 800x600, I suspect that you may be falling back to MMIO instead of a linear framebuffer. Try moving to 1024x768 and see if that helps. Regards, Anthony Liguori> > When booting the same domVT in runlevel 5 without ever attempting to connect to it with > vncviewer, qemu-dm fluctuated from 1-3% while the host was booting and then settled to 0.2% after 4-5 mins. Assuming that the boot process was complete, I launched vncviewer and qemu-dm jumped up to a 40-44% range again. > > Lastly I started the same host in runlevel 3 in pure text mode without any > vga parameters on the kernel line, connected to the virtual machine with > vncviewer, and watched the machine boot until a login prompt appeared. At > this point, top showed qemu-dm as using 0% of the cpu. Logging into the > virtual machine and running ''startx'' however sent qemu-dm back up to a 40- > 44% range. > > Why is qemu-dm consuming so much of the cpu? Why is qemu-dm still high > even when I close my vncviewer? > > Tommie, > Xen Test Team > Unisys > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Anthony,>You want to set it to NULL after a qemu_del_timer only in the connection >teardown code.Setting "ts->timer=NULL" and blocking access to this pointer in qemu_mod_timer seemed to resolve the issue. When I close the vnc window qemu-dm drops to 0%. Opening and closing a connection to the guest with vncviewer doesn''t seem to change the way time is being managed in the virtual machine. Is this the right fix for this problem? Signed-off-by: Tommie McAfee <tommie.mcafee@unisys.com> diff -r 8273f730371b tools/ioemu/vl.c --- a/tools/ioemu/vl.c Tue Aug 29 12:23:11 2006 +0100 +++ b/tools/ioemu/vl.c Wed Aug 30 13:03:01 2006 -0400 @@ -731,6 +731,8 @@ void qemu_mod_timer(QEMUTimer *ts, int64 { QEMUTimer **pt, *t; + if(!ts) + return; qemu_del_timer(ts); /* add the timer in the sorted list */ diff -r 8273f730371b tools/ioemu/vnc.c --- a/tools/ioemu/vnc.c Tue Aug 29 12:23:11 2006 +0100 +++ b/tools/ioemu/vnc.c Wed Aug 30 13:03:01 2006 -0400 @@ -626,6 +626,7 @@ static int vnc_client_io_error(VncState buffer_reset(&vs->input); buffer_reset(&vs->output); vs->need_update = 0; + vs->timer=NULL; return 0; } return ret; -------------------------- -----Original Message----- From: Anthony Liguori [mailto:aliguori@cs.utexas.edu] Sent: Wed 8/30/2006 5:05 PM To: McAfee, Tommie M Cc: Anthony Liguori; xen-devel@lists.xensource.com Subject: Re: qemu-dm cpu utilization McAfee, Tommie M wrote:> Anthony, > > >> If you delete the timer (and set vs->timer = NULL) in >> vnc_client_io_error(), does it reduce the usage? >> > > setting vs->timer=NULL causes qemu-dm to segfaultYou want to set it to NULL after a qemu_del_timer only in the connection teardown code.> According to strace, the timers account for a lot of qemu-dm''s activity. > Is this polling a bit excessive? >I don''t fully understand the performance of qemu bits in qemu-dm. Christian could probably offer a better insight especially since I think he changed the normal timers a bit (at least the GUI timer). When completely idle, qemu normally won''t occupy that much CPU (often less than a few percent). I''d expect it to be a little higher with qemu-dm. Regards, Anthony Liguori> Tommie McAfee > Xen-Testing > > > > -----Original Message----- > From: Anthony Liguori [mailto:anthony@codemonkey.ws] > Sent: Tue 8/29/2006 7:43 PM > To: McAfee, Tommie M > Subject: Re: qemu-dm cpu utilization > > McAfee, Tommie M wrote: > >> Using an 8-way ES7000 with 32GB of ram, I''m running xen at changset 11217 >> with dom0_mem=3092M as a kernel parameter on a x86_64 hyperviser. >> >> It appears that qemu is adding too much overhead to >> virtual machines to display graphics. >> If I boot a sles9sp3 domVT in runlevel 5 with vga=0x314 for a graphicalboot, and immediately use >> vncviewer simply to watch the virtual machine as it boots, >> qemu-dm consumes 40% cpu as soon as vncviewer is launched and remains that way throughout this entire process without settling down when the login screen appears. >> Even If I close the console that''s viewing the virtual host, qemu-dm remains at 40-44%. >> > > Almost everything in the VNC code resets when a guest disconnected > except for the timer. If a client disconnects, the timer routine should > be almost a nop but it''s possible that it''s somehow causing qemu-dm to > do something funky. > > If you delete the timer (and set vs->timer = NULL) in > vnc_client_io_error(), does it reduce the usage? > > BTW, if you''re using 800x600, I suspect that you may be falling back to > MMIO instead of a linear framebuffer. Try moving to 1024x768 and see if > that helps. > > Regards, > > Anthony Liguori > > >> When booting the same domVT in runlevel 5 without ever attempting to connect to it with >> vncviewer, qemu-dm fluctuated from 1-3% while the host was booting and then settled to 0.2% after 4-5 mins. Assuming that the boot process was complete, I launched vncviewer and qemu-dm jumped up to a 40-44% range again. >> >> Lastly I started the same host in runlevel 3 in pure text mode without any >> vga parameters on the kernel line, connected to the virtual machine with >> vncviewer, and watched the machine boot until a login prompt appeared. At >> this point, top showed qemu-dm as using 0% of the cpu. Logging into the >> virtual machine and running ''startx'' however sent qemu-dm back up to a 40- >> 44% range. >> >> Why is qemu-dm consuming so much of the cpu? Why is qemu-dm still high >> even when I close my vncviewer? >> >> Tommie, >> Xen Test Team >> Unisys >> >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >> > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel