Xen devel - Mar 2007 - [PATCH][HVM] remove qemu shadow_vram patch for performance

remove qemu shadow_vram patch and force a whole screen update each time for 
performance.

W/O this patch, there is huge performance drop in HVM domain when adding other 
guest(windows or linux with xwindow).

shadow_vram_revert.patch - revert the shadow_vram patch
shadow_vram_force_update.patch - explictly redraw screen each time

Signed-off-by: Liang, Kan <kan.liang@intel.com>
Signed-off-by: Zhai, Edwin <edwin.zhai@intel.com>

-- 
best rgds,
edwin



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 15/3/07 03:30, "Zhai, Edwin" <edwin.zhai@intel.com> wrote:
> remove qemu shadow_vram patch and force a whole screen update each time for
> performance.
> 
> W/O this patch, there is huge performance drop in HVM domain when adding
other
> guest(windows or linux with xwindow).
> 
> shadow_vram_revert.patch - revert the shadow_vram patch
> shadow_vram_force_update.patch - explictly redraw screen each time
How can updating the whole screen 30 times a second be faster than the
memcmp() that we do currently?

 -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, Mar 15, 2007 at 10:50:13AM +0000, Keir Fraser
wrote:
> 
> 
> 
> On 15/3/07 03:30, "Zhai, Edwin" <edwin.zhai@intel.com>
wrote:
> 
> > remove qemu shadow_vram patch and force a whole screen update each
time for
> > performance.
> > 
> > W/O this patch, there is huge performance drop in HVM domain when
adding other
> > guest(windows or linux with xwindow).
> > 
> > shadow_vram_revert.patch - revert the shadow_vram patch
> > shadow_vram_force_update.patch - explictly redraw screen each time
> 
> How can updating the whole screen 30 times a second be faster than the
> memcmp() that we do currently?
as far as i can tell, the bottle neck is that orig method does memcmp and memcpy
byte by byte. furthermore, orig method can void a update by multiple memcmp only
if all bytes are equal, which is in the minority.

there is no doubt we need a vram dirty for qemu, but current one is not the 
best. we can make a new one in future by shadow or something else.

thanks,
> 
>  -- Keir
> 
-- 
best rgds,
edwin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir, do you think this patch is OK?
-Xin 
>-----Original Message-----
>From: xen-devel-bounces@lists.xensource.com 
>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhai, Edwin
>Sent: Friday, March 16, 2007 11:04 AM
>To: Keir Fraser
>Cc: Ian Pratt; xen-devel@lists.xensource.com; Zhai, Edwin
>Subject: Re: [Xen-devel] [PATCH][HVM] remove qemu shadow_vram 
>patch forperformance
>
>On Thu, Mar 15, 2007 at 10:50:13AM +0000, Keir Fraser wrote:
>> 
>> 
>> 
>> On 15/3/07 03:30, "Zhai, Edwin" <edwin.zhai@intel.com>
wrote:
>> 
>> > remove qemu shadow_vram patch and force a whole screen 
>update each time for
>> > performance.
>> > 
>> > W/O this patch, there is huge performance drop in HVM 
>domain when adding other
>> > guest(windows or linux with xwindow).
>> > 
>> > shadow_vram_revert.patch - revert the shadow_vram patch
>> > shadow_vram_force_update.patch - explictly redraw screen each time
>> 
>> How can updating the whole screen 30 times a second be 
>faster than the
>> memcmp() that we do currently?
>
>as far as i can tell, the bottle neck is that orig method does 
>memcmp and memcpy 
>byte by byte. furthermore, orig method can void a update by 
>multiple memcmp only 
>if all bytes are equal, which is in the minority.
>
>there is no doubt we need a vram dirty for qemu, but current 
>one is not the 
>best. we can make a new one in future by shadow or something else.
>
>thanks,
>
>> 
>>  -- Keir
>> 
>
>-- 
>best rgds,
>edwin
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I punted on this one. :-) Christian wasn''t sure about it either, hence
it''s
not been checked in.

 -- Keir

On 20/3/07 08:38, "Li, Xin B" <xin.b.li@intel.com> wrote:
> Keir, do you think this patch is OK?
> -Xin 
> 
>> -----Original Message-----
>> From: xen-devel-bounces@lists.xensource.com
>> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhai, Edwin
>> Sent: Friday, March 16, 2007 11:04 AM
>> To: Keir Fraser
>> Cc: Ian Pratt; xen-devel@lists.xensource.com; Zhai, Edwin
>> Subject: Re: [Xen-devel] [PATCH][HVM] remove qemu shadow_vram
>> patch forperformance
>> 
>> On Thu, Mar 15, 2007 at 10:50:13AM +0000, Keir Fraser wrote:
>>> 
>>> 
>>> 
>>> On 15/3/07 03:30, "Zhai, Edwin"
<edwin.zhai@intel.com> wrote:
>>> 
>>>> remove qemu shadow_vram patch and force a whole screen
>> update each time for
>>>> performance.
>>>> 
>>>> W/O this patch, there is huge performance drop in HVM
>> domain when adding other
>>>> guest(windows or linux with xwindow).
>>>> 
>>>> shadow_vram_revert.patch - revert the shadow_vram patch
>>>> shadow_vram_force_update.patch - explictly redraw screen each
time
>>> 
>>> How can updating the whole screen 30 times a second be
>> faster than the
>>> memcmp() that we do currently?
>> 
>> as far as i can tell, the bottle neck is that orig method does
>> memcmp and memcpy
>> byte by byte. furthermore, orig method can void a update by
>> multiple memcmp only
>> if all bytes are equal, which is in the minority.
>> 
>> there is no doubt we need a vram dirty for qemu, but current
>> one is not the 
>> best. we can make a new one in future by shadow or something else.
>> 
>> thanks,
>> 
>>> 
>>>  -- Keir
>>> 
>> 
>> -- 
>> best rgds,
>> edwin
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

The major problem this patch is trying to fix is the performance drop
when multiple HVM guests are running, and I think we need a better
solution for this.
-Xin 
>-----Original Message-----
>From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk] 
>Sent: Tuesday, March 20, 2007 4:48 PM
>To: Li, Xin B; Zhai, Edwin; Keir Fraser
>Cc: Ian Pratt; xen-devel@lists.xensource.com
>Subject: Re: [Xen-devel] [PATCH][HVM] remove qemu shadow_vram 
>patch forperformance
>
>I punted on this one. :-) Christian wasn''t sure about it 
>either, hence it''s
>not been checked in.
>
> -- Keir
>
>On 20/3/07 08:38, "Li, Xin B" <xin.b.li@intel.com> wrote:
>
>> Keir, do you think this patch is OK?
>> -Xin 
>> 
>>> -----Original Message-----
>>> From: xen-devel-bounces@lists.xensource.com
>>> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of 
>Zhai, Edwin
>>> Sent: Friday, March 16, 2007 11:04 AM
>>> To: Keir Fraser
>>> Cc: Ian Pratt; xen-devel@lists.xensource.com; Zhai, Edwin
>>> Subject: Re: [Xen-devel] [PATCH][HVM] remove qemu shadow_vram
>>> patch forperformance
>>> 
>>> On Thu, Mar 15, 2007 at 10:50:13AM +0000, Keir Fraser wrote:
>>>> 
>>>> 
>>>> 
>>>> On 15/3/07 03:30, "Zhai, Edwin"
<edwin.zhai@intel.com> wrote:
>>>> 
>>>>> remove qemu shadow_vram patch and force a whole screen
>>> update each time for
>>>>> performance.
>>>>> 
>>>>> W/O this patch, there is huge performance drop in HVM
>>> domain when adding other
>>>>> guest(windows or linux with xwindow).
>>>>> 
>>>>> shadow_vram_revert.patch - revert the shadow_vram patch
>>>>> shadow_vram_force_update.patch - explictly redraw screen
each time
>>>> 
>>>> How can updating the whole screen 30 times a second be
>>> faster than the
>>>> memcmp() that we do currently?
>>> 
>>> as far as i can tell, the bottle neck is that orig method does
>>> memcmp and memcpy
>>> byte by byte. furthermore, orig method can void a update by
>>> multiple memcmp only
>>> if all bytes are equal, which is in the minority.
>>> 
>>> there is no doubt we need a vram dirty for qemu, but current
>>> one is not the 
>>> best. we can make a new one in future by shadow or something else.
>>> 
>>> thanks,
>>> 
>>>> 
>>>>  -- Keir
>>>> 
>>> 
>>> -- 
>>> best rgds,
>>> edwin
>>> 
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 20/3/07 09:36, "Li, Xin B" <xin.b.li@intel.com> wrote:
> The major problem this patch is trying to fix is the performance drop
> when multiple HVM guests are running, and I think we need a better
> solution for this.
>From my own reading of the code I would have thought that by removing that
SSE memcmp we''d be replacing periodic SSE block memcmp with periodic
pixel-by-pixel comparison (which is what I think we end up doing if we get
past the SSE memcmp?). I''m thinking particularly of the VNC side of
things
rather than SDL here, by the way: you guys use SDL most of the time, right?

 -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>-----Original Message-----
>From: Keir Fraser [mailto:keir@xensource.com] 
>Sent: Tuesday, March 20, 2007 5:45 PM
>To: Li, Xin B; Zhai, Edwin; Keir Fraser
>Cc: Ian Pratt; xen-devel@lists.xensource.com
>Subject: Re: [Xen-devel] [PATCH][HVM] remove qemu shadow_vram 
>patch forperformance
>
>On 20/3/07 09:36, "Li, Xin B" <xin.b.li@intel.com> wrote:
>
>> The major problem this patch is trying to fix is the performance drop
>> when multiple HVM guests are running, and I think we need a better
>> solution for this.
>
>From my own reading of the code I would have thought that by removing
that
>SSE memcmp we''d be replacing periodic SSE block memcmp with
periodic
>pixel-by-pixel comparison (which is what I think we end up doing if we
get
>past the SSE memcmp?). I''m thinking particularly of the VNC 
>side of things rather than SDL here, 
> by the way: you guys use SDL most of the time, right?
Yes :-)
-Xin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Mar 20, 2007 at 09:45:28AM +0000, Keir Fraser
wrote:
> On 20/3/07 09:36, "Li, Xin B" <xin.b.li@intel.com> wrote:
> 
> > The major problem this patch is trying to fix is the performance drop
> > when multiple HVM guests are running, and I think we need a better
> > solution for this.
> 
> >From my own reading of the code I would have thought that by removing
that
> SSE memcmp we''d be replacing periodic SSE block memcmp with
periodic
> pixel-by-pixel comparison (which is what I think we end up doing if we get
> past the SSE memcmp?). I''m thinking particularly of the VNC side
of things
> rather than SDL here, by the way: you guys use SDL most of the time, right?
our developer prefer SDL, but our performance data shows that VNS also has such 
issue, although is a little better than SDL.
> 
>  -- Keir
> 
-- 
best rgds,
edwin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> On 20/3/07 09:36, "Li, Xin B" <xin.b.li@intel.com> wrote:
>
>> The major problem this patch is trying to fix is the performance drop
>> when multiple HVM guests are running, and I think we need a better
>> solution for this.
>
>> From my own reading of the code I would have thought that by
>> removing that
> SSE memcmp we''d be replacing periodic SSE block memcmp with
periodic
> pixel-by-pixel comparison (which is what I think we end up doing if
> we get past the SSE memcmp?). I''m thinking particularly of the VNC
> side of things rather than SDL here, by the way: you guys use SDL
> most of the time, right?
>
Keir:
        I confirmed with Don that this memcmp will happen no matter the
Qemu
window is active or not (i.e. the screen is covered by other Windows).
So to
answer the question Anthony and people have, using memcmp all Qemu
window
need to do periodic memory comparation and may have many page fault
when accessing to shadow vram buffer used for comparation, and thus
eat a lot of CPU cycle and system bus cycles. Without the memcmp,
only the top Qemu window (usually most other windows are hidden)
will do full screen update, while others won''t.
        In benchmark where we run 8 VMs, it is very obvious that we will
benefit from no memcmp. Given that using shadow page table to indicate
a dirty page needs to solve the gpn or mfn to multiple gva backlink
issue,
pulling in this solution for 3.0.5 will increase HVM scalability
obviously.
thanks,eddie

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> 
> 
> On 15/3/07 03:30, "Zhai, Edwin" <edwin.zhai@intel.com>
wrote:
> 
>> remove qemu shadow_vram patch and force a whole screen update each time
for
>> performance.
>>
>> W/O this patch, there is huge performance drop in HVM domain when
adding other
>> guest(windows or linux with xwindow).
>>
>> shadow_vram_revert.patch - revert the shadow_vram patch
>> shadow_vram_force_update.patch - explictly redraw screen each time
> 
> How can updating the whole screen 30 times a second be faster than the
> memcmp() that we do currently?
It really depends.  The VNC display already has a minimization mechanism 
so doign the memcmp() in the vga driver doesn''t help at all.

For SDL, when using X, it''s going to be doing an XShmImage so the 
difference is only in the size of update (no data is transferred to the 
X server).  If the X server is double buffering the framebuffer (which I 
have to assume it''s doing), then you''re just paying the cost
of a
memcpy() in the X server that you would normally pay in qemu-dm.

However, if you''re forwarding SDL over X, or for some reason your X 
server isn''t using XShmImage, this patch will slow things down
considerably.

Perhaps this should be made a configuration option?

Regards,

Anthony Liguori
>  -- Keir

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> 
> 
> On 15/3/07 03:30, "Zhai, Edwin" <edwin.zhai@intel.com>
wrote:
> 
>> remove qemu shadow_vram patch and force a whole screen update each time
for
>> performance.
>>
>> W/O this patch, there is huge performance drop in HVM domain when
adding other
>> guest(windows or linux with xwindow).
>>
>> shadow_vram_revert.patch - revert the shadow_vram patch
>> shadow_vram_force_update.patch - explictly redraw screen each time
> 
> How can updating the whole screen 30 times a second be faster than the
> memcmp() that we do currently?
It really depends.  The VNC display already has a minimization mechanism 
so doign the memcmp() in the vga driver doesn''t help at all.

For SDL, when using X, it''s going to be doing an XShmImage so the 
difference is only in the size of update (no data is transferred to the 
X server).  If the X server is double buffering the framebuffer (which I 
have to assume it''s doing), then you''re just paying the cost
of a
memcpy() in the X server that you would normally pay in qemu-dm.

However, if you''re forwarding SDL over X, or for some reason your X 
server isn''t using XShmImage, this patch will slow things down
considerably.

Perhaps this should be made a configuration option?

Regards,

Anthony Liguori
>  -- Keir

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 3/20/07, Keir Fraser <Keir.Fraser@cl.cam.ac.uk>
wrote:
> I punted on this one. :-) Christian wasn''t sure about it either,
hence it''s
> not been checked in.
My main concern is that with the patch we will be doing the transform
from vga ram into the display memory everytime (vga_draw_line, some
versions are quite expensive) and I don''t see how that could be faster
than doing the sse2 optimized compare and copy?

When you test this, are the displays active or not, i.e. is the
display changing a lot or are they at the screen saver?

Maybe if the SDL frontend notices that its window is obscured, it
should tell the rest of the code that a scan is not required...

Same for VNC...

    christian
>
>  -- Keir
>
> On 20/3/07 08:38, "Li, Xin B" <xin.b.li@intel.com> wrote:
>
> > Keir, do you think this patch is OK?
> > -Xin
> >
> >> -----Original Message-----
> >> From: xen-devel-bounces@lists.xensource.com
> >> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhai,
Edwin
> >> Sent: Friday, March 16, 2007 11:04 AM
> >> To: Keir Fraser
> >> Cc: Ian Pratt; xen-devel@lists.xensource.com; Zhai, Edwin
> >> Subject: Re: [Xen-devel] [PATCH][HVM] remove qemu shadow_vram
> >> patch forperformance
> >>
> >> On Thu, Mar 15, 2007 at 10:50:13AM +0000, Keir Fraser wrote:
> >>>
> >>>
> >>>
> >>> On 15/3/07 03:30, "Zhai, Edwin"
<edwin.zhai@intel.com> wrote:
> >>>
> >>>> remove qemu shadow_vram patch and force a whole screen
> >> update each time for
> >>>> performance.
> >>>>
> >>>> W/O this patch, there is huge performance drop in HVM
> >> domain when adding other
> >>>> guest(windows or linux with xwindow).
> >>>>
> >>>> shadow_vram_revert.patch - revert the shadow_vram patch
> >>>> shadow_vram_force_update.patch - explictly redraw screen
each time
> >>>
> >>> How can updating the whole screen 30 times a second be
> >> faster than the
> >>> memcmp() that we do currently?
> >>
> >> as far as i can tell, the bottle neck is that orig method does
> >> memcmp and memcpy
> >> byte by byte. furthermore, orig method can void a update by
> >> multiple memcmp only
> >> if all bytes are equal, which is in the minority.
> >>
> >> there is no doubt we need a vram dirty for qemu, but current
> >> one is not the
> >> best. we can make a new one in future by shadow or something else.
> >>
> >> thanks,
> >>
> >>>
> >>>  -- Keir
> >>>
> >>
> >> --
> >> best rgds,
> >> edwin
> >>
> >> _______________________________________________
> >> Xen-devel mailing list
> >> Xen-devel@lists.xensource.com
> >> http://lists.xensource.com/xen-devel
> >>
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Christian Limpach wrote:
> On 3/20/07, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
>> I punted on this one. :-) Christian wasn''t sure about it
either,
>> hence it''s not been checked in.
>
> My main concern is that with the patch we will be doing the transform
> from vga ram into the display memory everytime (vga_draw_line, some
> versions are quite expensive) and I don''t see how that could be
faster
> than doing the sse2 optimized compare and copy?
>
> When you test this, are the displays active or not, i.e. is the
> display changing a lot or are they at the screen saver?
>
> Maybe if the SDL frontend notices that its window is obscured, it
> should tell the rest of the code that a scan is not required...
>
> Same for VNC...
>
Christian:
        It doesn''t matter if the screen is changing contents or not.
Comparing shadow vram with vram already consumes 2X memory bus
cycle. In case of 8 VMs, we get 16X memory bus cycle (and a lot
of cache flush , page fault for shadow vram and flush of other TLBs).
 While w/o memcmp, we only got 1 full screen update.
        Given that in multi core environment, different processor
running
different Qemu Windows will compete the memory access. So it seems
very nature the compare approach will become a bottle neck when VM #
increases.
        BTW, We didn''t see performance issue with single VM either, and
in multiple VM environment, most Qemu windows are usually hidden.

        thanks, eddie

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel