tinnycloud
2010-Dec-17 03:24 UTC
[Xen-devel] Win2003 got blue Screen when ballooning with serial out put: "Iomem mapping not permitted"
Hi Steven:
I was trying to make use of balloon driver on my win2003 HVM Guests,
but confront a Guest crash problem with blue screen.
The serial output is much like
(XEN)grant_table.c:555:d0 Iomem mapping not permitted
ffffffffffffffff
My pvdriver changeset is 823. From
http://xenbits.xensource.com/ext/win-pvdrivers.hg
Xen is 4.0.0
I''ve done quite a lot experiments to locate the crash, and finally
make it easy to reproduce.
Here is the situation:
1) Start several Win2003 Guest with 2G memory to occupy host memory
till from xm info | grep free to see xen has only almost 3G memory
2) Start another Guest with 2G memory, say domain X, and set its
target to be 512M, after balloon, dom X will has only 512M, and xen has 2.5G
memory
3) Start another Guest with 2G memory, then xen will have only 0.5G
memory left
4) At this time, I set the domain X target to 2G (use xenstore-write
/local/domain/x/memory/target 2097152)
5) domain X got blue screen at once. With code says 0x0000007f
(0x0000000d, 0x00000000, 0x00000000, 0x00000000)"
So, in short, when a domain wants more memory, it deflates its
balloon, but if Xen doesn''t enough memory, the domain will crash.
I''ve go through
http://lists.xensource.com/archives/html/xen-devel/2010-01/msg01230.html
http://lists.xensource.com/archives/html/xen-devel/2010-02/msg00074.html
it looks like the crash is because windows crash dumper go through
those balloon out pages
and from the discuss it would be harmless, but why I got this crash?
Could you kindly offer me some help, thanks.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Paul Durrant
2010-Dec-17 08:53 UTC
RE: [Xen-devel] Win2003 got blue Screen when ballooning with serial out put: "Iomem mapping not permitted"
If the frontend GPL PV storport driver is not ignoring block write failures in crashdump mode then you are unlikely the get a good dump from the ballooned down VM. As for the original crash, I guess the balloon driver is just not handling the populate hypercall failure gracefully. Paul> -----Original Message----- > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel- > bounces@lists.xensource.com] On Behalf Of tinnycloud > Sent: 17 December 2010 03:24 > To: Steven Smith; xen-devel@lists.xensource.com > Cc: james.harper@bendigoit.com.au; Keir Fraser; JBeulich@novell.com > Subject: [Xen-devel] Win2003 got blue Screen when ballooning with > serial out put: "Iomem mapping not permitted" > > Hi Steven: > > I was trying to make use of balloon driver on my win2003 HVM > Guests, > but confront a Guest crash problem with blue screen. > The serial output is much like > (XEN)grant_table.c:555:d0 Iomem mapping not permitted > ffffffffffffffff > > My pvdriver changeset is 823. From > http://xenbits.xensource.com/ext/win-pvdrivers.hg > Xen is 4.0.0 > > I''ve done quite a lot experiments to locate the crash, and > finally > make it easy to reproduce. > Here is the situation: > > 1) Start several Win2003 Guest with 2G memory to occupy host > memory > till from xm info | grep free to see xen has only almost 3G memory > 2) Start another Guest with 2G memory, say domain X, and set > its > target to be 512M, after balloon, dom X will has only 512M, and xen > has 2.5G > memory > 3) Start another Guest with 2G memory, then xen will have > only 0.5G > memory left > 4) At this time, I set the domain X target to 2G (use > xenstore-write > /local/domain/x/memory/target 2097152) > 5) domain X got blue screen at once. With code says 0x0000007f > (0x0000000d, 0x00000000, 0x00000000, 0x00000000)" > > So, in short, when a domain wants more memory, it deflates > its > balloon, but if Xen doesn''t enough memory, the domain will crash. > > I''ve go through > http://lists.xensource.com/archives/html/xen-devel/2010- > 01/msg01230.html > > http://lists.xensource.com/archives/html/xen-devel/2010- > 02/msg00074.html > > it looks like the crash is because windows crash dumper go > through > those balloon out pages > and from the discuss it would be harmless, but why I got this > crash? > > Could you kindly offer me some help, thanks. > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
James Harper
2010-Dec-17 09:20 UTC
RE: [Xen-devel] Win2003 got blue Screen when ballooning with serial out put: "Iomem mapping not permitted"
> > If the frontend GPL PV storport driver is not ignoring block writefailures in> crashdump mode then you are unlikely the get a good dump from theballooned> down VM.It does, because I had to balloon down a few pages for migration to work (probably an old bug by now). I did put a limit as to how many failures it was prepared to put up with though.> As for the original crash, I guess the balloon driver is just nothandling the> populate hypercall failure gracefully. >Quite likely. James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
tinnycloud
2010-Dec-17 09:39 UTC
re: [Xen-devel] Win2003 got blue Screen when ballooning with serial out put: "Iomem mapping not permitted"
Hi Paul:
You are right. Thanks.
From the code PV driver, the exception handle is not finished by
James.
KdPrint((__DRIVER_NAME " Calling
HYPERVISOR_memory_op(XENMEM_populate_physmap) - pfn_count = %d\n",
pfn_count));
ret = HYPERVISOR_memory_op(xpdd, XENMEM_populate_physmap,
&reservation);
ExFreePoolWithTag(pfns, XENPCI_POOL_TAG);
KdPrint((__DRIVER_NAME " populated %d pages\n", ret));
/* TODO: what do we do if less than the required number of pages
were populated??? can this happen??? *
I will first try to implement myself.
Also, that would be great appreciation if someone could offer this
patch.
-----mail-----
: Paul Durrant [mailto:Paul.Durrant@citrix.com]
sent: 2010年12月17日 16:53
to: tinnycloud; Steven Smith; xen-devel@lists.xensource.com
cc: james.harper@bendigoit.com.au; Keir Fraser; JBeulich@novell.com
Subject: RE: [Xen-devel] Win2003 got blue Screen when ballooning with serial
out put: "Iomem mapping not permitted"
If the frontend GPL PV storport driver is not ignoring block write failures
in crashdump mode then you are unlikely the get a good dump from the
ballooned down VM.
As for the original crash, I guess the balloon driver is just not handling
the populate hypercall failure gracefully.
Paul
> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> bounces@lists.xensource.com] On Behalf Of tinnycloud
> Sent: 17 December 2010 03:24
> To: Steven Smith; xen-devel@lists.xensource.com
> Cc: james.harper@bendigoit.com.au; Keir Fraser; JBeulich@novell.com
> Subject: [Xen-devel] Win2003 got blue Screen when ballooning with
> serial out put: "Iomem mapping not permitted"
>
> Hi Steven:
>
> I was trying to make use of balloon driver on my win2003 HVM
> Guests,
> but confront a Guest crash problem with blue screen.
> The serial output is much like
> (XEN)grant_table.c:555:d0 Iomem mapping not permitted
> ffffffffffffffff
>
> My pvdriver changeset is 823. From
> http://xenbits.xensource.com/ext/win-pvdrivers.hg
> Xen is 4.0.0
>
> I''ve done quite a lot experiments to locate the crash, and
> finally
> make it easy to reproduce.
> Here is the situation:
>
> 1) Start several Win2003 Guest with 2G memory to occupy host
> memory
> till from xm info | grep free to see xen has only almost 3G memory
> 2) Start another Guest with 2G memory, say domain X, and set
> its
> target to be 512M, after balloon, dom X will has only 512M, and xen
> has 2.5G
> memory
> 3) Start another Guest with 2G memory, then xen will have
> only 0.5G
> memory left
> 4) At this time, I set the domain X target to 2G (use
> xenstore-write
> /local/domain/x/memory/target 2097152)
> 5) domain X got blue screen at once. With code says 0x0000007f
> (0x0000000d, 0x00000000, 0x00000000, 0x00000000)"
>
> So, in short, when a domain wants more memory, it deflates
> its
> balloon, but if Xen doesn''t enough memory, the domain will crash.
>
> I''ve go through
> http://lists.xensource.com/archives/html/xen-devel/2010-
> 01/msg01230.html
>
> http://lists.xensource.com/archives/html/xen-devel/2010-
> 02/msg00074.html
>
> it looks like the crash is because windows crash dumper go
> through
> those balloon out pages
> and from the discuss it would be harmless, but why I got this
> crash?
>
> Could you kindly offer me some help, thanks.
>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
tinnycloud
2010-Dec-20 02:06 UTC
re: [Xen-devel] Win2003 got blue Screen when ballooning with serial out put: "Iomem mapping not permitted"
Hi James: Thanks for your help. Beside not handling populating page failure, it seems like PV driver hasn''t support POD yet. Another question is, in Linux Balloon driver, every time balloon 2MB, while in windows PV driver, the balloon unit is 1MB, Is there anything make this difference? Thanks. ----- ----- From: James Harper [mailto:james.harper@bendigoit.com.au] date: 20101217 17:20 tp: Paul Durrant; tinnycloud; Steven Smith; xen-devel@lists.xensource.com cc: Keir Fraser; JBeulich@novell.com subject: RE: [Xen-devel] Win2003 got blue Screen when ballooning with serial out put: "Iomem mapping not permitted"> > If the frontend GPL PV storport driver is not ignoring block writefailures in> crashdump mode then you are unlikely the get a good dump from theballooned> down VM.It does, because I had to balloon down a few pages for migration to work (probably an old bug by now). I did put a limit as to how many failures it was prepared to put up with though.> As for the original crash, I guess the balloon driver is just nothandling the> populate hypercall failure gracefully. >Quite likely. James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
xiaoyun.maoxy
2011-Jan-05 13:11 UTC
[Xen-devel] Could the page sharing inside domain? (memory_sharing on xen)
Hi Keir: One of the features listed as having been added to Xen 4 is page sharing for HVM domUs. Pity is it has been disabled since buggy. Well I tried to make it work these days. After go through the code and get thoroughly concept of page sharing, I''ve been able to setup test environment, but confronted domU crash. One abnormal output from serial is (XEN) mm.c:859:d0 Error getting mfn 2df15f (pfn fffffffffffffffe) from L1 entry 80000002df15f627 for l1e_owner=0, pg_owner=4 (XEN) mm.c:859:d0 Error getting mfn 2df160 (pfn fffffffffffffffe) from L1 entry 80000002df160627 for l1e_owner=0, pg_owner=4 Later, I found this is related to page sharing inside domain itself. That is when a domain X first read IO sectors Y, it will nominate the gref into xen(that is build a hash entry in xen) , after IO completion, it will build a hash entry in tapdisk2 hash table. But later domain X read same sectors again, it can find entry in both hash tables, and page sharing works, one page freed. Then I can see above serial output. So my question is: Is page sharing legal inside domain? Thanks. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
HI James:
I''ve noticed the TODO tag in xenpci/xenpci_fdo.c, line 294
290 KdPrint((__DRIVER_NAME " Calling
HYPERVISOR_memory_op(XENMEM_populate_physmap) - pfn_count = %d\n",
pfn_count));
291 ret = HYPERVISOR_memory_op(xpdd, XENMEM_populate_physmap,
&reservation);
292 ExFreePoolWithTag(pfns, XENPCI_POOL_TAG);
293 KdPrint((__DRIVER_NAME " populated %d pages\n",
ret));
294 /* TODO: what do we do if less than the required number of
pages were populated??? can this happen??? */
Could we fix it in linux balloon driver way?
When no enough pfn_count got, give memory back to xen.
What do you think?
Thanks.
/* TODO: what do we do if less than the required number of pages were
populated??? can this happen??? */
/* fixed! */
if(ret < (ULONG)pfn_count){
if(ret > 0){
/* We hit the Xen hard limit: reprobe. */
reservation.nr_extents = ret;
ret = HYPERVISOR_memory_op(xpdd,
XENMEM_decrease_reservation, &reservation);
KdPrint((__DRIVER_NAME " decreased %d pages\n",
ret));
}
mdl->Next = head;
head = mdl;
ExFreePoolWithTag(pfns, XENPCI_POOL_TAG);
break;
}
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Hi: We''ve confronted the windows 2003 Blue screen when using balloon driver. The blue screen has the information of "NO_PAGES_AVAILABLE", and Technical information: ***STOP: 0x0000004D (0x0002A8FB,0x0002A8FB,0x00000000,0x00000000) In fact we are doing the stress test on 24 windows2003 HVM on a 24G, 16core host. In dom0, there is a daemon process (balloond) will give memory back to the VM who acclaim memory. The balloond will ensure every VM at least has 512M memory. Each VM is started with memory=512 maxmem=2048 Inside each of the VM, there is two processes eat memory, each of which will eat 1G memory. And the page file size configured on C:\ is init size 1536M, max size 3072M, which I think even If balloond process not give back the memory back to VM, the VM can has 512M+3072M = 3.5G virtual memory. Am I right?, if so , since our eat memory process only consume 2G memory, h how could "NO_PAGES_AVAILABLE" happen? Also, since we enable VM writes its memory status into xenstore, and when blue screen, it has only has 972KB memory. Any comments? Many thanks. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Hi: > > We''ve confronted the windows 2003 Blue screen when using balloon > driver. > > The blue screen has the information of "NO_PAGES_AVAILABLE", and > Technical information: > ***STOP: 0x0000004D(0x0002A8FB,0x0002A8FB,0x00000000,0x00000000)> > In fact we are doing the stress test on 24 windows2003 HVM on a24G,> 16core host. > In dom0, there is a daemon process (balloond) will give memoryback> to the VM who acclaim memory. > The balloond will ensure every VM at least has 512M memory. > > Each VM is started with memory=512 maxmem=2048 > Inside each of the VM, there is two processes eat memory, eachof> which will eat 1G memory. > And the page file size configured on C:\ is init size 1536M, max > size 3072M, which I think even > If balloond process not give back the memory back to VM, the VMcan> has 512M+3072M = 3.5G virtual > memory. > > Am I right?, if so , since our eat memory process only consume2G> memory, h > how could "NO_PAGES_AVAILABLE" happen? > > Also, since we enable VM writes its memory status into xenstore,and> when blue screen, it has only has 972KB memory. > > Any comments? Many thanks. >You''ve put myself (I wrote GPLPV) and George Dunlap (from Citrix) in the CC list but you don''t say if you are using GPLPV drivers or Citrix PV drivers. If you are using GPLPV drivers let me know and I''ll get some more information about the problem from you. If you aren''t using GPLPV drivers then I don''t think I can be of much assistance... Thanks James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > HI James: > > I''ve noticed the TODO tag in xenpci/xenpci_fdo.c, line 294 > > 290 KdPrint((__DRIVER_NAME " Calling > HYPERVISOR_memory_op(XENMEM_populate_physmap) - pfn_count = %d\n", > pfn_count)); > 291 ret = HYPERVISOR_memory_op(xpdd,XENMEM_populate_physmap,> &reservation); > 292 ExFreePoolWithTag(pfns, XENPCI_POOL_TAG); > 293 KdPrint((__DRIVER_NAME " populated %d pages\n",ret));> 294 /* TODO: what do we do if less than the required numberof> pages were populated??? can this happen??? */ > > > Could we fix it in linux balloon driver way? > When no enough pfn_count got, give memory back to xen. > What do you think? > Thanks.I guess that would work, but then it would keep trying to decrease again every second. Maybe that''s what we want but I''d rather put in some sort of backoff in there eg retry in one second then two seconds then 4 seconds etc up to a maximum of 1 minute or so, to reduce the noise. Are you seeing this happen? I''m guessing it''s because xen doesn''t have enough pages for you or because you''ve ballooned up above the maxmem limit or something... James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ye, You are right, it happens when xen doesn''t have enough pages for the VM. And windows will crash now. In memory overcommit, If there are many VMs on a physical host, and in dom0 use a daemon for memory managment between VMS, it is possible that xen has no pages for the VM needs memory. So I think , it is better to fix it. Thanks.> Subject: RE: TODO in windows balloon driver > Date: Sat, 26 Feb 2011 23:17:54 +1100 > From: james.harper@bendigoit.com.au > To: tinnycloud@hotmail.com; xen-devel@lists.xensource.com > > > > > HI James: > > > > I''ve noticed the TODO tag in xenpci/xenpci_fdo.c, line 294 > > > > 290 KdPrint((__DRIVER_NAME " Calling > > HYPERVISOR_memory_op(XENMEM_populate_physmap) - pfn_count = %d\n", > > pfn_count)); > > 291 ret = HYPERVISOR_memory_op(xpdd, > XENMEM_populate_physmap, > > &reservation); > > 292 ExFreePoolWithTag(pfns, XENPCI_POOL_TAG); > > 293 KdPrint((__DRIVER_NAME " populated %d pages\n", > ret)); > > 294 /* TODO: what do we do if less than the required number > of > > pages were populated??? can this happen??? */ > > > > > > Could we fix it in linux balloon driver way? > > When no enough pfn_count got, give memory back to xen. > > What do you think? > > Thanks. > > I guess that would work, but then it would keep trying to decrease again > every second. Maybe that''s what we want but I''d rather put in some sort > of backoff in there eg retry in one second then two seconds then 4 > seconds etc up to a maximum of 1 minute or so, to reduce the noise. > > Are you seeing this happen? I''m guessing it''s because xen doesn''t have > enough pages for you or because you''ve ballooned up above the maxmem > limit or something... > > James >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > Ye, You are right, it happens when xen doesn''t have enough pages forthe VM.> And windows will crash now. > > In memory overcommit, If there are many VMs on a physical host, and indom0> use a daemon for > memory managment between VMS, it is possible that xen has no pages forthe VM> needs memory. > > So I think , it is better to fix it. >I have put your fix in as well as the backoff delay I mentioned earlier, but I''m reviewing the ballooning stuff now. I can see one bug where I don''t do the initial balloon down in the case of PoD. I''ve fixed that, but I''m not doing it early enough, so for Windows 2008 32 bit with memory=512 and maxmem=1024, Windows still crashes and ''xm dmesg'' shows "Out of populate-on-demand memory!" just after my driver starts to load (in DriverEntry). It seems to work fine when memory=768 and maxmem=1024 but I still think it could crash on startup if you are unlucky. The only possible workaround is to do it in DriverEntry of xenpci but that will require a bit of a refactoring. I think it''s probably worth doing though. James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > > > > Ye, You are right, it happens when xen doesn''t have enough pages for > the VM. > > And windows will crash now. > > > > In memory overcommit, If there are many VMs on a physical host, andin> dom0 > > use a daemon for > > memory managment between VMS, it is possible that xen has no pagesfor> the VM > > needs memory. > > > > So I think , it is better to fix it. > > > > I have put your fix in as well as the backoff delay I mentionedearlier,> but I''m reviewing the ballooning stuff now. I can see one bug where I > don''t do the initial balloon down in the case of PoD. I''ve fixed that, > but I''m not doing it early enough, so for Windows 2008 32 bit with > memory=512 and maxmem=1024, Windows still crashes and ''xm dmesg'' shows > "Out of populate-on-demand memory!" just after my driver starts toload> (in DriverEntry). It seems to work fine when memory=768 andmaxmem=1024> but I still think it could crash on startup if you are unlucky. > > The only possible workaround is to do it in DriverEntry of xenpci but > that will require a bit of a refactoring. I think it''s probably worth > doing though. >I''ve just pushed a partial fix for this. The initial balloon-down is now done during DriverEntry and I''m able to boot with memory=256 and maxmem=1024 without any problems (other than that Windows doesn''t run particularly well with 256MB of memory :) I haven''t worked out a way to pass the memory reserved in DriverEntry back to the xenpci FDO yet though so it''s not particularly useful at this time as you can''t balloon up again. I''ll fix that soon now that I have the DriverEntry thing working. James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel