I''m looking at PoD support in GPLPV. There was a bug preventing the initial balloon down of DomU which I''ve now fixed, but it''s still a bit of a race with Windows to try and balloon down the pages before Windows tries to use them. Under Windows 2008 x32, memory=512 + maxmem=1024 is enough to crash the system just after my DriverEntry is called. Do other Windows PV drivers support PoD? Under what scenarios? I''m trying to figure out how achievable this is. Thanks James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
We tested a wide range of memory parameters for the Citrix PV drivers on XenServer. I just asked the guy in charge of that, and he said that they were able to reliably boot w2k8x32 with memory=128MiB and maxmem=1024MiB. I know we had to do make some changes to move the balloon driver allocation further back in the boot process to make that happen; but I''m not familiar with the details. Paul would be the person to ask about that. -George On Sun, Feb 27, 2011 at 1:47 AM, James Harper <james.harper@bendigoit.com.au> wrote:> I''m looking at PoD support in GPLPV. There was a bug preventing the > initial balloon down of DomU which I''ve now fixed, but it''s still a bit > of a race with Windows to try and balloon down the pages before Windows > tries to use them. Under Windows 2008 x32, memory=512 + maxmem=1024 is > enough to crash the system just after my DriverEntry is called. > > Do other Windows PV drivers support PoD? Under what scenarios? I''m > trying to figure out how achievable this is. > > Thanks > > James > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I actually have plans to push it earlier because we balloon down quite late at the moment (off the back of the START IRP in the top level bus driver). We are reliant upon zero-page sweeping code in Xen to save us from guest crashing up to that point. Paul> -----Original Message----- > From: dunlapg@gmail.com [mailto:dunlapg@gmail.com] On Behalf Of > George Dunlap > Sent: 28 February 2011 11:59 > To: James Harper > Cc: xen devel; Paul Durrant > Subject: Re: [Xen-devel] PoD in other (not GPLPV) drivers > > We tested a wide range of memory parameters for the Citrix PV > drivers on XenServer. I just asked the guy in charge of that, and > he said that they were able to reliably boot w2k8x32 with > memory=128MiB and maxmem=1024MiB. > > I know we had to do make some changes to move the balloon driver > allocation further back in the boot process to make that happen; but > I''m not familiar with the details. Paul would be the person to ask > about that. > > -George > > On Sun, Feb 27, 2011 at 1:47 AM, James Harper > <james.harper@bendigoit.com.au> wrote: > > I''m looking at PoD support in GPLPV. There was a bug preventing > the > > initial balloon down of DomU which I''ve now fixed, but it''s still > a > > bit of a race with Windows to try and balloon down the pages > before > > Windows tries to use them. Under Windows 2008 x32, memory=512 + > > maxmem=1024 is enough to crash the system just after my > DriverEntry is called. > > > > Do other Windows PV drivers support PoD? Under what scenarios? I''m > > trying to figure out how achievable this is. > > > > Thanks > > > > James > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > I actually have plans to push it earlier because we balloon down quitelate at> the moment (off the back of the START IRP in the top level busdriver). We are> reliant upon zero-page sweeping code in Xen to save us from guestcrashing up> to that point. >I''ve modified GPLPV to balloon down at DriverEntry time, which seems to be early enough. Prior to that, memory=128 and maxmem=1024 was enough to cause a crash under 2008, basically as soon as I tried to access the registry in DriverEntry. My drivers are using WDF and are therefore loading after the KMDF framework which is going to use additional resources. My backup plan is to write a WDM driver that loads even earlier than that and does the allocation, passing it to the real PV drivers later on, although my concern there is that Windows may not like memory allocated by one driver being freed by another... I''ve never heard of ''zero-page sweeping code'' before... is that a way of xen reallocating a previously touched page if it contains all 0''s if we want a page beyond our allocation limit? That might explain why my initial balloon down is so slow! I can tell windows to not zero pages before it gives them to me when I do the initial balloon down... what are your thoughts on that? Although it''s unlikely at boot time, in theory they could contain sensitive information and I''m supposed to zero them before handing them back to xen according to the docs. James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
James, Indeed, zero page sweeping is code inside xen that runs if the PoD cache is exhausted. It refills the cache with pages that are zeroed out (replacing them with pod entries in the p2m). Your initial balloon down may well be slow if you (or the windows kernel) touches the page before you hand it back to xen, since the pod code will have to populate each page in the p2m as it is touched. Paul> -----Original Message----- > From: James Harper [mailto:james.harper@bendigoit.com.au] > Sent: 28 February 2011 12:29 > To: Paul Durrant; George Dunlap > Cc: xen devel > Subject: RE: [Xen-devel] PoD in other (not GPLPV) drivers > > > > > I actually have plans to push it earlier because we balloon down > quite > late at > > the moment (off the back of the START IRP in the top level bus > driver). We are > > reliant upon zero-page sweeping code in Xen to save us from guest > crashing up > > to that point. > > > > I''ve modified GPLPV to balloon down at DriverEntry time, which seems > to be early enough. Prior to that, memory=128 and maxmem=1024 was > enough to cause a crash under 2008, basically as soon as I tried to > access the registry in DriverEntry. My drivers are using WDF and are > therefore loading after the KMDF framework which is going to use > additional resources. My backup plan is to write a WDM driver that > loads even earlier than that and does the allocation, passing it to > the real PV drivers later on, although my concern there is that > Windows may not like memory allocated by one driver being freed by > another... > > I''ve never heard of ''zero-page sweeping code'' before... is that a > way of xen reallocating a previously touched page if it contains all > 0''s if we want a page beyond our allocation limit? That might > explain why my initial balloon down is so slow! I can tell windows > to not zero pages before it gives them to me when I do the initial > balloon down... what are your thoughts on that? Although it''s > unlikely at boot time, in theory they could contain sensitive > information and I''m supposed to zero them before handing them back > to xen according to the docs. > > James_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Just for the public record: I believe that I have upstreamed all of the PoD fixes and improvements that we had in the XenServer tree with one exception: There are a series of patches to improve the "emergency zero-page sweep" behavior in Xen, particularly for pretty large (>24GiB) guests. This is because: * They were really ugly; hacks upon hacks, not suitable for upstreaming * As far as I know, they''re performance enhancements only, not correctness ones. It doesn''t sound to me like the zero-page sweeps should be involved in your issue, James. But if you (or anyone) would like the patches, I''d be happy to send them along for you to try. (And for the shy, or archaeologists looking at this long after I''m gone from Citrix, they can be found on the XenServer source ISOs.) I''m not sure they''ll apply cleanly to 4.1, but that code hasn''t had a lot of changes, so you should be able to work out how they apply. As part of my work to port XenServer to 4.1, I''m going to rewrite the patches, and I''ll upstream them at that time. -George On Mon, Feb 28, 2011 at 1:41 PM, Paul Durrant <Paul.Durrant@citrix.com> wrote:> James, > > Indeed, zero page sweeping is code inside xen that runs if the PoD cache is exhausted. It refills the cache with pages that are zeroed out (replacing them with pod entries in the p2m). Your initial balloon down may well be slow if you (or the windows kernel) touches the page before you hand it back to xen, since the pod code will have to populate each page in the p2m as it is touched. > > Paul > >> -----Original Message----- >> From: James Harper [mailto:james.harper@bendigoit.com.au] >> Sent: 28 February 2011 12:29 >> To: Paul Durrant; George Dunlap >> Cc: xen devel >> Subject: RE: [Xen-devel] PoD in other (not GPLPV) drivers >> >> > >> > I actually have plans to push it earlier because we balloon down >> quite >> late at >> > the moment (off the back of the START IRP in the top level bus >> driver). We are >> > reliant upon zero-page sweeping code in Xen to save us from guest >> crashing up >> > to that point. >> > >> >> I''ve modified GPLPV to balloon down at DriverEntry time, which seems >> to be early enough. Prior to that, memory=128 and maxmem=1024 was >> enough to cause a crash under 2008, basically as soon as I tried to >> access the registry in DriverEntry. My drivers are using WDF and are >> therefore loading after the KMDF framework which is going to use >> additional resources. My backup plan is to write a WDM driver that >> loads even earlier than that and does the allocation, passing it to >> the real PV drivers later on, although my concern there is that >> Windows may not like memory allocated by one driver being freed by >> another... >> >> I''ve never heard of ''zero-page sweeping code'' before... is that a >> way of xen reallocating a previously touched page if it contains all >> 0''s if we want a page beyond our allocation limit? That might >> explain why my initial balloon down is so slow! I can tell windows >> to not zero pages before it gives them to me when I do the initial >> balloon down... what are your thoughts on that? Although it''s >> unlikely at boot time, in theory they could contain sensitive >> information and I''m supposed to zero them before handing them back >> to xen according to the docs. >> >> James > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > Just for the public record: I believe that I have upstreamed all of > the PoD fixes and improvements that we had in the XenServer tree with > one exception: There are a series of patches to improve the "emergency > zero-page sweep" behavior in Xen, particularly for pretty large > (>24GiB) guests. This is because: > * They were really ugly; hacks upon hacks, not suitable forupstreaming> * As far as I know, they''re performance enhancements only, notcorrectness> ones. > > It doesn''t sound to me like the zero-page sweeps should be involved in > your issue, James. But if you (or anyone) would like the patches, I''d > be happy to send them along for you to try. (And for the shy, or > archaeologists looking at this long after I''m gone from Citrix, they > can be found on the XenServer source ISOs.) I''m not sure they''ll > apply cleanly to 4.1, but that code hasn''t had a lot of changes, so > you should be able to work out how they apply. > > As part of my work to port XenServer to 4.1, I''m going to rewrite the > patches, and I''ll upstream them at that time. >Even without pre-zeroing the memory under Windows, it is still pretty slow doing the initial balloon down. Even though I ask windows not to zero fill the page (therefore populating it), windows may still be touching it though. But even then I give it straight back to xen so in hindsight I think the zero page sweep shouldn''t get invoked. For testing I''m trying some pretty extreme numbers though, eg maxmem=16384, memory=512, so ballooning down 15.5GB of memory is never going to be that fast. I now suspect that the delay is in the Windows API. When I balloon down during normal runtime it is about 10x slower than ballooning up. James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > We tested a wide range of memory parameters for the Citrix PV drivers > on XenServer. I just asked the guy in charge of that, and he said > that they were able to reliably boot w2k8x32 with memory=128MiB and > maxmem=1024MiB. > > I know we had to do make some changes to move the balloon driver > allocation further back in the boot process to make that happen; but > I''m not familiar with the details. Paul would be the person to ask > about that. >Is there a way for DomU to get an interrupt when an unpopulated page is hit when the DomU has gone past its initial limit? That would allow me to at least write a message to the log and crash usefully if I can''t do anything else about it. James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, 2011-02-28 at 22:20 +0000, James Harper wrote:> Is there a way for DomU to get an interrupt when an unpopulated page is > hit when the DomU has gone past its initial limit? That would allow me > to at least write a message to the log and crash usefully if I can''t do > anything else about it.At the moment there''s no mechanism for this. Doing so would be a bit tricky, as most of the places which do gfn_to_mfn() translations aren''t designed to handle failures gracefully (although that may have changed since some of the guest paging stuff was introduced). If we were to do this at some point in the future, what do you think you''d like the interface to look like? (i.e., deliver NMI? MCE? a different but well-known interrupt? &c) -George _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > On Mon, 2011-02-28 at 22:20 +0000, James Harper wrote: > > Is there a way for DomU to get an interrupt when an unpopulated page is > > hit when the DomU has gone past its initial limit? That would allow me > > to at least write a message to the log and crash usefully if I can't do > > anything else about it. > > At the moment there's no mechanism for this. Doing so would be a bit > tricky, as most of the places which do gfn_to_mfn() translations aren't > designed to handle failures gracefully (although that may have changed > since some of the guest paging stuff was introduced). > > If we were to do this at some point in the future, what do you think > you'd like the interface to look like? (i.e., deliver NMI? MCE? a > different but well-known interrupt? &c) >I don't know that there is a sactioned way to intercept an NMI under windows. An MCE would do the trick probably. Basically I want the user to know that their VM crashed because it hit the PoD limit, rather than have no idea why it crashed. Most of the time if a user gets a crash I ask for a copy of /var/log/xen/qemu-dm-<domu>.log, so if the message appeared there then that would be fine with me too. I was under the impression that xen would blow the DomU up if it overstepped its PoD limit, but now I'm not so sure as I've seen windows BSoD with lots of different bug check codes when it happens. What should happen? I think that instant death is the only sensible outcome. James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> -----Original Message----- > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel- > bounces@lists.xensource.com] On Behalf Of James Harper > Sent: 03 March 2011 10:22 > To: George Dunlap > Cc: George Dunlap; xen devel > Subject: RE: [Xen-devel] PoD in other (not GPLPV) drivers > > > > > On Mon, 2011-02-28 at 22:20 +0000, James Harper wrote: > > > Is there a way for DomU to get an interrupt when an unpopulated > page > > > is hit when the DomU has gone past its initial limit? That would > > > allow me to at least write a message to the log and crash > usefully > > > if I can't do anything else about it. > > > > At the moment there's no mechanism for this. Doing so would be a > bit > > tricky, as most of the places which do gfn_to_mfn() translations > > aren't designed to handle failures gracefully (although that may > have > > changed since some of the guest paging stuff was introduced). > > > > If we were to do this at some point in the future, what do you > think > > you'd like the interface to look like? (i.e., deliver NMI? MCE? a > > different but well-known interrupt? &c) > > > > I don't know that there is a sactioned way to intercept an NMI under > windows.KeRegisterNmiCallback() See http://msdn.microsoft.com/en-us/library/ff553116%28v=vs.85%29.aspx> An MCE would do the trick probably. Basically I want the > user to know that their VM crashed because it hit the PoD limit, > rather than have no idea why it crashed. Most of the time if a user > gets a crash I ask for a copy of /var/log/xen/qemu-dm-<domu>.log, so > if the message appeared there then that would be fine with me too. > > I was under the impression that xen would blow the DomU up if it > overstepped its PoD limit, but now I'm not so sure as I've seen > windows BSoD with lots of different bug check codes when it happens. > What should happen? I think that instant death is the only sensible > outcome. > > James >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel