We recently found that FreeBSD 8.0 guest failed to install and boot on Xen. The reason was that FreeBSD detected clflush feature and invoked this instruction to flush MMIO space. This caused a page fault; but x86_emulate.c failed to emulate this instruction (not supported). As a result, a page fault was detected inside FreeBSD. A similar issue was reported earlier. http://lists.xensource.com/archives/html/xen-devel/2010-03/msg00362.html I created a patch which enables clflush emulation. I have verified that this patch solves FreeBSD issue. This patch returns immediately without doing anything. Note that we don''t have to intercept clflush in SVM. So the only time we have page_fault for clflush instruction is when guest VM flushes MMIO space. In this case, do we need to send this command over to QEMU? I didn''t do it in this patch anyway. Best, -Wei _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 15/04/2010 17:38, "Wei Huang" <wei.huang2@amd.com> wrote:> I created a patch which enables clflush emulation. I have verified that > this patch solves FreeBSD issue. This patch returns immediately without > doing anything. Note that we don''t have to intercept clflush in SVM. So > the only time we have page_fault for clflush instruction is when guest > VM flushes MMIO space. In this case, do we need to send this command > over to QEMU? I didn''t do it in this patch anyway.I don''t think we need to anything since all guest reads/writes to the mmio space will get intercepted to qemu, and hence qemu''s view of the space is coherent with the guest with no need for special handling of CLFLUSH. Still the emulator should call out and let the caller decide what to do. We could safely turn CLFLUSH into WBINVD (i.e., call the existing wbinvd hook). Do you know if these CLFLUSH emulations happen often or only very rarely? That might help us decide how smart we need to be about emulating CLFLUSH (i.e., add a hook specifically for clflush, separately from wbinvd). -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
WBINVD is a bit heavy since it flushes all caches. I saw 1170 times of CLFLUSH during FreeBSD booting. After guest boots, 0 was observed. 1170 is not huge though. -Wei Keir Fraser wrote:> On 15/04/2010 17:38, "Wei Huang" <wei.huang2@amd.com> wrote: > > >> I created a patch which enables clflush emulation. I have verified that >> this patch solves FreeBSD issue. This patch returns immediately without >> doing anything. Note that we don''t have to intercept clflush in SVM. So >> the only time we have page_fault for clflush instruction is when guest >> VM flushes MMIO space. In this case, do we need to send this command >> over to QEMU? I didn''t do it in this patch anyway. >> > > I don''t think we need to anything since all guest reads/writes to the mmio > space will get intercepted to qemu, and hence qemu''s view of the space is > coherent with the guest with no need for special handling of CLFLUSH. > > Still the emulator should call out and let the caller decide what to do. We > could safely turn CLFLUSH into WBINVD (i.e., call the existing wbinvd hook). > Do you know if these CLFLUSH emulations happen often or only very rarely? > That might help us decide how smart we need to be about emulating CLFLUSH > (i.e., add a hook specifically for clflush, separately from wbinvd). > > -- Keir > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Our WBINVD emulation doesn''t do anything unless the guest has some devices passed through to it. I think that will do for now then. Thanks, Keir On 15/04/2010 18:25, "Wei Huang" <wei.huang2@amd.com> wrote:> WBINVD is a bit heavy since it flushes all caches. I saw 1170 times of > CLFLUSH during FreeBSD booting. After guest boots, 0 was observed. 1170 > is not huge though. > > -Wei > > Keir Fraser wrote: >> On 15/04/2010 17:38, "Wei Huang" <wei.huang2@amd.com> wrote: >> >> >>> I created a patch which enables clflush emulation. I have verified that >>> this patch solves FreeBSD issue. This patch returns immediately without >>> doing anything. Note that we don''t have to intercept clflush in SVM. So >>> the only time we have page_fault for clflush instruction is when guest >>> VM flushes MMIO space. In this case, do we need to send this command >>> over to QEMU? I didn''t do it in this patch anyway. >>> >> >> I don''t think we need to anything since all guest reads/writes to the mmio >> space will get intercepted to qemu, and hence qemu''s view of the space is >> coherent with the guest with no need for special handling of CLFLUSH. >> >> Still the emulator should call out and let the caller decide what to do. We >> could safely turn CLFLUSH into WBINVD (i.e., call the existing wbinvd hook). >> Do you know if these CLFLUSH emulations happen often or only very rarely? >> That might help us decide how smart we need to be about emulating CLFLUSH >> (i.e., add a hook specifically for clflush, separately from wbinvd). >> >> -- Keir >> >> >> >> > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
See what you think of xen-unstable:21189. It is also a bit more precise in its decoding of CLFLUSH, as a member of Grp15. -- Keir On 15/04/2010 18:32, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:> Our WBINVD emulation doesn''t do anything unless the guest has some devices > passed through to it. I think that will do for now then. > > Thanks, > Keir > > On 15/04/2010 18:25, "Wei Huang" <wei.huang2@amd.com> wrote: > >> WBINVD is a bit heavy since it flushes all caches. I saw 1170 times of >> CLFLUSH during FreeBSD booting. After guest boots, 0 was observed. 1170 >> is not huge though. >> >> -Wei >> >> Keir Fraser wrote: >>> On 15/04/2010 17:38, "Wei Huang" <wei.huang2@amd.com> wrote: >>> >>> >>>> I created a patch which enables clflush emulation. I have verified that >>>> this patch solves FreeBSD issue. This patch returns immediately without >>>> doing anything. Note that we don''t have to intercept clflush in SVM. So >>>> the only time we have page_fault for clflush instruction is when guest >>>> VM flushes MMIO space. In this case, do we need to send this command >>>> over to QEMU? I didn''t do it in this patch anyway. >>>> >>> >>> I don''t think we need to anything since all guest reads/writes to the mmio >>> space will get intercepted to qemu, and hence qemu''s view of the space is >>> coherent with the guest with no need for special handling of CLFLUSH. >>> >>> Still the emulator should call out and let the caller decide what to do. We >>> could safely turn CLFLUSH into WBINVD (i.e., call the existing wbinvd hook). >>> Do you know if these CLFLUSH emulations happen often or only very rarely? >>> That might help us decide how smart we need to be about emulating CLFLUSH >>> (i.e., add a hook specifically for clflush, separately from wbinvd). >>> >>> -- Keir >>> >>> >>> >>> >> >> > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I can confirm that 21189 fixed FreeBSD issue. Thanks, -Wei -----Original Message----- From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] Sent: Thursday, April 15, 2010 12:50 PM To: Huang2, Wei Cc: ''xen-devel@lists.xensource.com'' Subject: Re: [Xen-devel] [PATCH] clflush instruction emulation See what you think of xen-unstable:21189. It is also a bit more precise in its decoding of CLFLUSH, as a member of Grp15. -- Keir On 15/04/2010 18:32, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:> Our WBINVD emulation doesn''t do anything unless the guest has some devices > passed through to it. I think that will do for now then. > > Thanks, > Keir > > On 15/04/2010 18:25, "Wei Huang" <wei.huang2@amd.com> wrote: > >> WBINVD is a bit heavy since it flushes all caches. I saw 1170 times of >> CLFLUSH during FreeBSD booting. After guest boots, 0 was observed. 1170 >> is not huge though. >> >> -Wei >> >> Keir Fraser wrote: >>> On 15/04/2010 17:38, "Wei Huang" <wei.huang2@amd.com> wrote: >>> >>> >>>> I created a patch which enables clflush emulation. I have verified that >>>> this patch solves FreeBSD issue. This patch returns immediately without >>>> doing anything. Note that we don''t have to intercept clflush in SVM. So >>>> the only time we have page_fault for clflush instruction is when guest >>>> VM flushes MMIO space. In this case, do we need to send this command >>>> over to QEMU? I didn''t do it in this patch anyway. >>>> >>> >>> I don''t think we need to anything since all guest reads/writes to the mmio >>> space will get intercepted to qemu, and hence qemu''s view of the space is >>> coherent with the guest with no need for special handling of CLFLUSH. >>> >>> Still the emulator should call out and let the caller decide what to do. We >>> could safely turn CLFLUSH into WBINVD (i.e., call the existing wbinvd hook). >>> Do you know if these CLFLUSH emulations happen often or only very rarely? >>> That might help us decide how smart we need to be about emulating CLFLUSH >>> (i.e., add a hook specifically for clflush, separately from wbinvd). >>> >>> -- Keir >>> >>> >>> >>> >> >> > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel