A user has pointed out a problem with GPLPV under Xen 4.2 when using PoD. I''m using the difference between XENMEM_maximum_reservation and XENMEM_current_reservation to tell me how much I should balloon down to account for PoD, but when the user has ballooned down to 1G (from 4Gb or 8GB), GPLPV logs the following: 13005008825593: XenPCI XENMEM_maximum_reservation = 262400 13005008825593: XenPCI XENMEM_current_reservation = 262136 13005008825609: XenPCI Trying to give 1056 KB (1 MB) to Xen What is the correct way to tell how much PoD memory there is under 4.2? Am I doing it wrong? I balloon down as early as possible (before xenbus starts) to avoid windows going over its limit so I''m hoping I can determine the size of PoD memory just via hypercalls. Thanks James
On Mon, Feb 11, 2013 at 12:03 AM, James Harper < james.harper@bendigoit.com.au> wrote:> A user has pointed out a problem with GPLPV under Xen 4.2 when using PoD. > I''m using the difference between XENMEM_maximum_reservation and > XENMEM_current_reservation to tell me how much I should balloon down to > account for PoD, but when the user has ballooned down to 1G (from 4Gb or > 8GB), GPLPV logs the following: > > 13005008825593: XenPCI XENMEM_maximum_reservation = 262400 > 13005008825593: XenPCI XENMEM_current_reservation = 262136 > 13005008825609: XenPCI Trying to give 1056 KB (1 MB) to Xen > > What is the correct way to tell how much PoD memory there is under 4.2? Am > I doing it wrong? > > I balloon down as early as possible (before xenbus starts) to avoid > windows going over its limit so I''m hoping I can determine the size of PoD > memory just via hypercalls. >You shouldn''t have to know anything specifically about PoD -- you should just look at the balloon target for the guest written in xenstore. The idea was as much as possible for the toolstack and Xen to work together to make it transparent to the balloon driver, in part because we expected to be running legacy drivers. The Citrix PV drivers don''t do anything special wrt PoD memory. (Paul, please correct me if I''m wrong.) WRT timing and xenbus, a couple of things: * Windows does a scrub of all memory at start-of-day. Especially on multiple-vcpu systems, we were unable to start the balloon process early enough to win the race against the scrubber, so we had to have ways of "reclaiming" zeroed pages for the PoD pool. What this means is that it''s not a matter of Windows *touching* memory, but of windows *dirtying* memory that will lead to a problem. * So there is a minimum amount of memory Windows needs to be able to make it to the stage where the balloon driver can run. When XenServer first implemented DMC, the team did extensive testing to determine minimum values above which Windows never crashed or hung, and (as I understand it) baked those into the xapi toolstack as a "seatbelt" to prevent users from setting the value too low. Not sure if that helps in your particular case -- I think 1G was within the limit, but I could be wrong. Dave, any comments? -George _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
> > On Mon, Feb 11, 2013 at 12:03 AM, James Harper > <james.harper@bendigoit.com.au> wrote: > > A user has pointed out a problem with GPLPV under Xen 4.2 when > using PoD. I''m using the difference between > XENMEM_maximum_reservation and XENMEM_current_reservation to tell > me how much I should balloon down to account for PoD, but when the user > has ballooned down to 1G (from 4Gb or 8GB), GPLPV logs the following: > > 13005008825593: XenPCI XENMEM_maximum_reservation > 262400 > 13005008825593: XenPCI XENMEM_current_reservation = 262136 > 13005008825609: XenPCI Trying to give 1056 KB (1 MB) to Xen > > What is the correct way to tell how much PoD memory there is under > 4.2? Am I doing it wrong? > > I balloon down as early as possible (before xenbus starts) to avoid > windows going over its limit so I''m hoping I can determine the size of PoD > memory just via hypercalls. > > > > You shouldn''t have to know anything specifically about PoD -- you should just > look at the balloon target for the guest written in xenstore. The idea was as > much as possible for the toolstack and Xen to work together to make it > transparent to the balloon driver, in part because we expected to be running > legacy drivers. The Citrix PV drivers don''t do anything special wrt PoD > memory. (Paul, please correct me if I''m wrong.)So I should just balloon down to the current_reservation figure right?> WRT timing and xenbus, a couple of things: > > * Windows does a scrub of all memory at start-of-day. Especially on multiple- > vcpu systems, we were unable to start the balloon process early enough to > win the race against the scrubber, so we had to have ways of "reclaiming" > zeroed pages for the PoD pool. What this means is that it''s not a matter of > Windows *touching* memory, but of windows *dirtying* memory that will > lead to a problem. > > * So there is a minimum amount of memory Windows needs to be able to > make it to the stage where the balloon driver can run. When XenServer first > implemented DMC, the team did extensive testing to determine minimum > values above which Windows never crashed or hung, and (as I understand it) > baked those into the xapi toolstack as a "seatbelt" to prevent users from > setting the value too low. > > Not sure if that helps in your particular case -- I think 1G was within the limit, > but I could be wrong. Dave, any comments? >I think I could go from 4GB down to 128MB reliably without crashing, although the resulting OS wasn''t particularly usable :) Thanks James
On 11/02/13 11:39, James Harper wrote:>> On Mon, Feb 11, 2013 at 12:03 AM, James Harper >> <james.harper@bendigoit.com.au> wrote: >> >> A user has pointed out a problem with GPLPV under Xen 4.2 when >> using PoD. I''m using the difference between >> XENMEM_maximum_reservation and XENMEM_current_reservation to tell >> me how much I should balloon down to account for PoD, but when the user >> has ballooned down to 1G (from 4Gb or 8GB), GPLPV logs the following: >> >> 13005008825593: XenPCI XENMEM_maximum_reservation >> 262400 >> 13005008825593: XenPCI XENMEM_current_reservation = 262136 >> 13005008825609: XenPCI Trying to give 1056 KB (1 MB) to Xen >> >> What is the correct way to tell how much PoD memory there is under >> 4.2? Am I doing it wrong? >> >> I balloon down as early as possible (before xenbus starts) to avoid >> windows going over its limit so I''m hoping I can determine the size of PoD >> memory just via hypercalls. >> >> >> >> You shouldn''t have to know anything specifically about PoD -- you should just >> look at the balloon target for the guest written in xenstore. The idea was as >> much as possible for the toolstack and Xen to work together to make it >> transparent to the balloon driver, in part because we expected to be running >> legacy drivers. The Citrix PV drivers don''t do anything special wrt PoD >> memory. (Paul, please correct me if I''m wrong.) > So I should just balloon down to the current_reservation figure right?...I don''t think so -- it looks like you''re getting that from a hypercall, not from xenstore. You want the normal balloon target value from xenstore. (But I might be confused -- I''m not super familiar with the technical details of the ballooning codepath, just the overall principles.)>> WRT timing and xenbus, a couple of things: >> >> * Windows does a scrub of all memory at start-of-day. Especially on multiple- >> vcpu systems, we were unable to start the balloon process early enough to >> win the race against the scrubber, so we had to have ways of "reclaiming" >> zeroed pages for the PoD pool. What this means is that it''s not a matter of >> Windows *touching* memory, but of windows *dirtying* memory that will >> lead to a problem. >> >> * So there is a minimum amount of memory Windows needs to be able to >> make it to the stage where the balloon driver can run. When XenServer first >> implemented DMC, the team did extensive testing to determine minimum >> values above which Windows never crashed or hung, and (as I understand it) >> baked those into the xapi toolstack as a "seatbelt" to prevent users from >> setting the value too low. >> >> Not sure if that helps in your particular case -- I think 1G was within the limit, >> but I could be wrong. Dave, any comments? >> > I think I could go from 4GB down to 128MB reliably without crashing, although the resulting OS wasn''t particularly usable :)I think what we found was that Windows required a certain amount of memory to *boot*. If you gave it less than that, then either one of two things would happen: * The processes dirtying RAM beat the balloon driver, and the guest crashed with a PoD-cache-empty error * The balloon driver beat the processes dirtying ram, and the guest hung; presumably some service somewhere waiting for some memory to "free up". In theory after boot you could reduce the memory below that threshold. But then what would you do if you needed to reboot the guest? You''d have to go grab the memory from somewhere else, which is the situation we were trying to avoid with having PoD in the first place. So the guy architecting DMC just made it a hard floor (again, as I understand it). -George
>>> On 11.02.13 at 12:40, George Dunlap <george.dunlap@eu.citrix.com> wrote: > On 11/02/13 11:39, James Harper wrote: >>> On Mon, Feb 11, 2013 at 12:03 AM, James Harper >>> <james.harper@bendigoit.com.au> wrote: >>> >>> A user has pointed out a problem with GPLPV under Xen 4.2 when >>> using PoD. I''m using the difference between >>> XENMEM_maximum_reservation and XENMEM_current_reservation to tell >>> me how much I should balloon down to account for PoD, but when the user >>> has ballooned down to 1G (from 4Gb or 8GB), GPLPV logs the following: >>> >>> 13005008825593: XenPCI XENMEM_maximum_reservation >>> 262400 >>> 13005008825593: XenPCI XENMEM_current_reservation = 262136 >>> 13005008825609: XenPCI Trying to give 1056 KB (1 MB) to Xen >>> >>> What is the correct way to tell how much PoD memory there is under >>> 4.2? Am I doing it wrong? >>> >>> I balloon down as early as possible (before xenbus starts) to avoid >>> windows going over its limit so I''m hoping I can determine the size of PoD >>> memory just via hypercalls. >>> >>> >>> >>> You shouldn''t have to know anything specifically about PoD -- you should just >>> look at the balloon target for the guest written in xenstore. The idea was > as >>> much as possible for the toolstack and Xen to work together to make it >>> transparent to the balloon driver, in part because we expected to be running >>> legacy drivers. The Citrix PV drivers don''t do anything special wrt PoD >>> memory. (Paul, please correct me if I''m wrong.) >> So I should just balloon down to the current_reservation figure right? > > ...I don''t think so -- it looks like you''re getting that from a > hypercall, not from xenstore. You want the normal balloon target value > from xenstore. (But I might be confused -- I''m not super familiar with > the technical details of the ballooning codepath, just the overall > principles.)As James said, this ought to run without xenbus having initialized (so it can be done as early as possible). Hence the need for doing this via hypercalls. The "classic" Xen balloon driver does so too, btw. Jan
> >> So I should just balloon down to the current_reservation figure right? > > > > ...I don''t think so -- it looks like you''re getting that from a > > hypercall, not from xenstore. You want the normal balloon target value > > from xenstore. (But I might be confused -- I''m not super familiar with > > the technical details of the ballooning codepath, just the overall > > principles.) > > As James said, this ought to run without xenbus having initialized > (so it can be done as early as possible). Hence the need for doing > this via hypercalls. The "classic" Xen balloon driver does so too, btw. >So is my mistake simply that I''m assuming that ''maximum reservation'' == ''total memory? It occurs to me that I don''t actually know how to get the total amount of memory from a windows driver... guess I''d better look it up! James
> -----Original Message----- > From: Jan Beulich [mailto:JBeulich@suse.com] > Sent: 12 February 2013 09:54 > To: James Harper; George Dunlap > Cc: Paul Durrant; Dave Scott; xen-devel@lists.xen.org > Subject: Re: [Xen-devel] PoD, 4.2, and current/maximum reservation > > >>> On 11.02.13 at 12:40, George Dunlap <george.dunlap@eu.citrix.com> > wrote: > > On 11/02/13 11:39, James Harper wrote: > >>> On Mon, Feb 11, 2013 at 12:03 AM, James Harper > >>> <james.harper@bendigoit.com.au> wrote: > >>> > >>> A user has pointed out a problem with GPLPV under Xen 4.2 when > >>> using PoD. I''m using the difference between > >>> XENMEM_maximum_reservation and XENMEM_current_reservation to > tell > >>> me how much I should balloon down to account for PoD, but when the > user > >>> has ballooned down to 1G (from 4Gb or 8GB), GPLPV logs the following: > >>> > >>> 13005008825593: XenPCI XENMEM_maximum_reservation > >>> 262400 > >>> 13005008825593: XenPCI XENMEM_current_reservation = 262136 > >>> 13005008825609: XenPCI Trying to give 1056 KB (1 MB) to Xen > >>> > >>> What is the correct way to tell how much PoD memory there is under > >>> 4.2? Am I doing it wrong? > >>> > >>> I balloon down as early as possible (before xenbus starts) to avoid > >>> windows going over its limit so I''m hoping I can determine the size of > PoD > >>> memory just via hypercalls. > >>> > >>> > >>> > >>> You shouldn''t have to know anything specifically about PoD -- you should > just > >>> look at the balloon target for the guest written in xenstore. The idea > was > > as > >>> much as possible for the toolstack and Xen to work together to make it > >>> transparent to the balloon driver, in part because we expected to be > running > >>> legacy drivers. The Citrix PV drivers don''t do anything special wrt PoD > >>> memory. (Paul, please correct me if I''m wrong.) > >> So I should just balloon down to the current_reservation figure right? > > > > ...I don''t think so -- it looks like you''re getting that from a > > hypercall, not from xenstore. You want the normal balloon target value > > from xenstore. (But I might be confused -- I''m not super familiar with > > the technical details of the ballooning codepath, just the overall > > principles.) > > As James said, this ought to run without xenbus having initialized > (so it can be done as early as possible). Hence the need for doing > this via hypercalls. The "classic" Xen balloon driver does so too, btw. >We just use xenbus in the Citrix PV drivers and rely on the zero page sweep code in Xen to prevent a memory scrub from crashing the domain. One problem with Windows can be SKU memory limits i.e. the OS cannot actually allocate from all the memory that is given to the domain so the target cannot necessarily be met. Paul
> > > > As James said, this ought to run without xenbus having initialized > > (so it can be done as early as possible). Hence the need for doing > > this via hypercalls. The "classic" Xen balloon driver does so too, btw. > > > > We just use xenbus in the Citrix PV drivers and rely on the zero page sweep > code in Xen to prevent a memory scrub from crashing the domain. >I was using the difference between current_reservation and maximum_reservation but maximum_reservation doesn''t appear to return the value in 4.2 that it did in 4.1 (based on one user report). current_reservation still appears to though so as long as I figure out how much physical memory is in the system (harder than it seems but a requirement even if you are using xenbus) I should be back in business.> One problem with Windows can be SKU memory limits i.e. the OS cannot > actually allocate from all the memory that is given to the domain so the target > cannot necessarily be met. >Like giving memory=2GB, maxmem=6GB to an XP32 system and wondering why things don''t work? I guess that''s a case of don''t do that :) Windows has too many secrets. James