Xen devel - Aug 2012 - qemu-xen-traditional: NOCACHE or CACHE_WB to open disk images for IDE

Hi list,

Recently I was debugging L2 guest slow booting issue in nested virtualization
environment (both L0 and L1 hypervisors are all Xen).
To boot a L2 Linux guest (RHEL6u2), it will need to wait more than 3 minutes
after grub loaded, I did some profile, and see guest is doing disk operations by
int13 BIOS procedure.

Even not consider the nested case, I saw there is a bug reporting normal VM
boots slower than before (actually both qcow and disk image), see:
http://bugzilla.xen.org/bugzilla/show_bug.cgi?id=1821
Therefore I think the boot delay is just much lengthened in L2 guest.

I root caused this issue to a change in qemu, and I saw there is a lot of
discussions on this topic. I didn''t see the final decision but later
the patch was checked in. Could anybody helps to revisit this commit and explain
the final decision?
http://lists.xen.org/archives/html/xen-devel/2012-03/msg02072.html

Thanks,
Dongxiao

Hi list,

Recently I was debugging L2 guest slow booting issue in nested virtualization
environment (both L0 and L1 hypervisors are all Xen).
To boot a L2 Linux guest (RHEL6u2), it will need to wait more than 3 minutes
after grub loaded, I did some profile, and see guest is doing disk operations by
int13 BIOS procedure.

Even not consider the nested case, I saw there is a bug reporting normal VM
boots slower than before (actually both qcow and disk image), see:
http://bugzilla.xen.org/bugzilla/show_bug.cgi?id=1821
Therefore I think the boot delay is just much lengthened in L2 guest.

I root caused this issue to a change in qemu, and I saw there is a lot of
discussions on this topic. I didn''t see the final decision but later
the patch was checked in. Could anybody helps to revisit this commit and explain
the final decision?
http://lists.xen.org/archives/html/xen-devel/2012-03/msg02072.html

Thanks,
Dongxiao

I attached the original patch here, which impact the performance.

Signed-off-by: Stefano Stabellini <stefano.stabellini@xxxxxxxxxxxxx>
---
 xenstore.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/xenstore.c b/xenstore.c
index 4c483e2..ac90366 100644
--- a/xenstore.c
+++ b/xenstore.c
@@ -643,7 +643,7 @@ void xenstore_parse_domain_config(int hvm_domid)
            }
             pstrcpy(bs->filename, sizeof(bs->filename), params);
 
-            flags = BDRV_O_CACHE_WB; /* snapshot and write-back */
+            flags = BDRV_O_NOCACHE;
             is_readonly = 0;
             if (pasprintf(&buf, "%s/mode", bpath) == -1)
                 continue;
-- 
1.7.2.5

Could somebody tell me the reason why this is finally checked in with
performance impact?

Thanks,
Dongxiao
> -----Original Message-----
> From: xen-devel-bounces@lists.xen.org
> [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of Xu, Dongxiao
> Sent: Friday, August 10, 2012 1:33 PM
> To: xen-devel@lists.xen.org
> Cc: Zhang, Yang Z; Ian Jackson; Ian Campbell; Stefano Stabellini
> Subject: [Xen-devel] qemu-xen-traditional: NOCACHE or CACHE_WB to open
> disk images for IDE
> 
> Hi list,
> 
> Recently I was debugging L2 guest slow booting issue in nested
virtualization
> environment (both L0 and L1 hypervisors are all Xen).
> To boot a L2 Linux guest (RHEL6u2), it will need to wait more than 3
minutes
> after grub loaded, I did some profile, and see guest is doing disk
operations by
> int13 BIOS procedure.
> 
> Even not consider the nested case, I saw there is a bug reporting normal VM
> boots slower than before (actually both qcow and disk image), see:
> http://bugzilla.xen.org/bugzilla/show_bug.cgi?id=1821
> Therefore I think the boot delay is just much lengthened in L2 guest.
> 
> I root caused this issue to a change in qemu, and I saw there is a lot of
> discussions on this topic. I didn''t see the final decision but
later the patch was
> checked in. Could anybody helps to revisit this commit and explain the
final
> decision?
> http://lists.xen.org/archives/html/xen-devel/2012-03/msg02072.html
> 
> Thanks,
> Dongxiao
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Xu, Dongxiao writes ("RE: [Xen-devel] qemu-xen-traditional: NOCACHE or
CACHE_WB to open disk images for IDE"):
> I attached the original patch here, which impact the performance.
Stefano, when you proposed what is now
1307e42a4b3c1102d75401bc0cffb4eb6c9b7a38 this you described it as a
major performance improvement.

Would you care to comment ?

Ian.

On Tue, 14 Aug 2012, Ian Jackson wrote:
> Xu, Dongxiao writes ("RE: [Xen-devel] qemu-xen-traditional: NOCACHE or
CACHE_WB to open disk images for IDE"):
> > I attached the original patch here, which impact the performance.
> 
> Stefano, when you proposed what is now
> 1307e42a4b3c1102d75401bc0cffb4eb6c9b7a38 this you described it as a
> major performance improvement.
> 
> Would you care to comment ?
QDISK needs NOCACHE, that''s why I wrote the original patch series.

I also thought that IDE also needs NOCACHE for safety but after a
lengthy discussion, we came up with the conclusion that WRITEBACK is
OK for IDE, see your message:

http://marc.info/?l=xen-devel&m=133311527009773

So I think we should revert 1307e42a4b3c1102d75401bc0cffb4eb6c9b7a38, I
don''t know why it was committed.

Also it seems to me that libxl is not specifying cache=writeback for
upstream QEMU, that means it is going to default to writethrough.
I''ll write a patch for that.

Stefano Stabellini writes ("RE: [Xen-devel] qemu-xen-traditional: NOCACHE
or CACHE_WB to open disk images for IDE"):
> I also thought that IDE also needs NOCACHE for safety but after a
> lengthy discussion, we came up with the conclusion that WRITEBACK is
> OK for IDE, see your message:
> 
> http://marc.info/?l=xen-devel&m=133311527009773
> 
> So I think we should revert 1307e42a4b3c1102d75401bc0cffb4eb6c9b7a38, I
> don''t know why it was committed.
I was probably confused.  Sorry.  I have reverted it.
> Also it seems to me that libxl is not specifying cache=writeback for
> upstream QEMU, that means it is going to default to writethrough.
> I''ll write a patch for that.
Right, thanks.

Ian.

> -----Original Message-----
> From: Ian Jackson [mailto:Ian.Jackson@eu.citrix.com]
> Sent: Tuesday, August 14, 2012 10:59 PM
> To: Stefano Stabellini
> Cc: Xu, Dongxiao; xen-devel@lists.xen.org; Zhang, Yang Z; Ian Campbell
> Subject: RE: [Xen-devel] qemu-xen-traditional: NOCACHE or CACHE_WB to
> open disk images for IDE
> 
> Stefano Stabellini writes ("RE: [Xen-devel] qemu-xen-traditional:
NOCACHE or
> CACHE_WB to open disk images for IDE"):
> > I also thought that IDE also needs NOCACHE for safety but after a
> > lengthy discussion, we came up with the conclusion that WRITEBACK is
> > OK for IDE, see your message:
> >
> > http://marc.info/?l=xen-devel&m=133311527009773
> >
> > So I think we should revert 1307e42a4b3c1102d75401bc0cffb4eb6c9b7a38,
> > I don''t know why it was committed.
> 
> I was probably confused.  Sorry.  I have reverted it.
Thanks, this helps me a lot.

Thanks,
Dongxiao
> 
> > Also it seems to me that libxl is not specifying cache=writeback for
> > upstream QEMU, that means it is going to default to writethrough.
> > I''ll write a patch for that.
> 
> Right, thanks.
> 
> Ian.