Xen devel - Jan 2010 - "Iomem mapping not permitted" during windows crash dump under GPLPV

I''ve recently noticed that my windows crash dumps fail at around 40-50%
under GPLPV. ''xm dmesg'' shows the following:

(XEN) grant_table.c:350:d0 Iomem mapping not permitted ffffffffffffffff
(domain 865)

At first I thought that the cause was just a bug in my grant ref code
but it just occurred to me that this could be happening when Windows
tries to write out the physical page I have mapped for some other xen
function. grant_table.c:350 is around here:

if ( !iomem_access_permitted(rd, frame, frame) )
{
  gdprintk(XENLOG_WARNING,
           "Iomem mapping not permitted %lx (domain %d)\n",
           frame, rd->domain_id);
  rc = GNTST_general_error;
  goto undo_out;
}

So is that likely to be the cause? I haven''t yet checked the pfn that
is
failing against the pages I''ve mapped for various things but the number
seems plausible. It used to work... is that check new under 3.4.1-ish?

And whats the solution? At this stage the only think I can think of is
that it might be reasonable to ignore a small amount of failed writes
during a crash dump... I haven''t yet researched if there is a way to
tell windows not to write out a given page of memory when doing a crash
dump.

Thanks

James

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

It''s not really an i/o page. All 1s means INVALID_MFN, which means the
p2m
lookup failed. I suppose Windows may think it is a valid pfn (e.g., an
emulated device?) but that kind of mem can''t be granted and never could
be.

 -- Keir

On 30/01/2010 05:26, "James Harper"
<james.harper@bendigoit.com.au> wrote:
> I''ve recently noticed that my windows crash dumps fail at around
40-50%
> under GPLPV. ''xm dmesg'' shows the following:
> 
> (XEN) grant_table.c:350:d0 Iomem mapping not permitted ffffffffffffffff
> (domain 865)
> 
> At first I thought that the cause was just a bug in my grant ref code
> but it just occurred to me that this could be happening when Windows
> tries to write out the physical page I have mapped for some other xen
> function. grant_table.c:350 is around here:
> 
> if ( !iomem_access_permitted(rd, frame, frame) )
> {
>   gdprintk(XENLOG_WARNING,
>            "Iomem mapping not permitted %lx (domain %d)\n",
>            frame, rd->domain_id);
>   rc = GNTST_general_error;
>   goto undo_out;
> }
> 
> So is that likely to be the cause? I haven''t yet checked the pfn
that is
> failing against the pages I''ve mapped for various things but the
number
> seems plausible. It used to work... is that check new under 3.4.1-ish?
> 
> And whats the solution? At this stage the only think I can think of is
> that it might be reasonable to ignore a small amount of failed writes
> during a crash dump... I haven''t yet researched if there is a way
to
> tell windows not to write out a given page of memory when doing a crash
> dump.
> 
> Thanks
> 
> James
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> 
> It''s not really an i/o page. All 1s means INVALID_MFN, which means
the
p2m
> lookup failed. I suppose Windows may think it is a valid pfn (e.g., an
> emulated device?) but that kind of mem can''t be granted and never
could be.
> 
That pfn is right in the middle of normal memory - 0x2c0f, 0x2749,
0x274a are the examples I have, and the DomU has 768MB allocated to it.

I''ll try and add some debug code to get the pfn back out of the grant
table when the error occurs, just in case a bug has caused the entry to
get corrupt.

So there is definitely nothing I could tell Xen to do with a DomU PFN
that would remove the p2m mapping?

Thanks

James

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 30/01/2010 08:30, "James Harper"
<james.harper@bendigoit.com.au> wrote:
> So there is definitely nothing I could tell Xen to do with a DomU PFN
> that would remove the p2m mapping?
If you balloon out a page, that would do it.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> On 30/01/2010 08:30, "James Harper"
<james.harper@bendigoit.com.au> wrote:
> 
>> So there is definitely nothing I could tell Xen to do with a DomU PFN
>> that would remove the p2m mapping?
> 
> If you balloon out a page, that would do it.
> 
Yes, it certainly would and I''ve seen it many times. I''m
fairly sure
that by just ignoring the failed block writes coming back from 
blkback/tap and carrying on you still end up with a usable MEMORY.DMP 
after reboot.

   Paul

-- 
==============================Paul Durrant, Software Engineer

Citrix Systems (R&D) Ltd.
First Floor, Building 101
Cambridge Science Park
Milton Road
Cambridge CB4 0FY
United Kingdom
==============================
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> 
> Keir Fraser wrote:
> > On 30/01/2010 08:30, "James Harper"
<james.harper@bendigoit.com.au>
wrote:
> >
> >> So there is definitely nothing I could tell Xen to do with a DomU
PFN
> >> that would remove the p2m mapping?
> >
> > If you balloon out a page, that would do it.
> >
> 
> Yes, it certainly would and I''ve seen it many times. I''m
fairly sure
> that by just ignoring the failed block writes coming back from
> blkback/tap and carrying on you still end up with a usable MEMORY.DMP
> after reboot.
> 
Yes, that works. I still eye the error messages with contempt though :)

James

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

James Harper wrote:
>> Keir Fraser wrote:
>>> On 30/01/2010 08:30, "James Harper"
<james.harper@bendigoit.com.au>
> wrote:
>>>> So there is definitely nothing I could tell Xen to do with a
DomU
> PFN
>>>> that would remove the p2m mapping?
>>> If you balloon out a page, that would do it.
>>>
>> Yes, it certainly would and I''ve seen it many times.
I''m fairly sure
>> that by just ignoring the failed block writes coming back from
>> blkback/tap and carrying on you still end up with a usable MEMORY.DMP
>> after reboot.
>>
> 
> Yes, that works. I still eye the error messages with contempt though :)
> 
If you wanted to be clean, I guess you could check the data buffer addr 
passed to you by Windows against a map of ballooned out pages and 
substitute a dummy buffer in this case... but then you have the 
''interesting'' problem of making the dump driver aware of that
map.

   Paul

-- 
==============================Paul Durrant, Software Engineer

Citrix Systems (R&D) Ltd.
First Floor, Building 101
Cambridge Science Park
Milton Road
Cambridge CB4 0FY
United Kingdom
==============================
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> James Harper wrote:
> >> Keir Fraser wrote:
> >>> On 30/01/2010 08:30, "James Harper"
<james.harper@bendigoit.com.au>
> > wrote:
> >>>> So there is definitely nothing I could tell Xen to do with
a DomU
> > PFN
> >>>> that would remove the p2m mapping?
> >>> If you balloon out a page, that would do it.
> >>>
> >> Yes, it certainly would and I''ve seen it many times.
I''m fairly
sure
> >> that by just ignoring the failed block writes coming back from
> >> blkback/tap and carrying on you still end up with a usable
MEMORY.DMP
> >> after reboot.
> >>
> >
> > Yes, that works. I still eye the error messages with contempt though
:)
> >
> 
> If you wanted to be clean, I guess you could check the data buffer
addr
> passed to you by Windows against a map of ballooned out pages and
> substitute a dummy buffer in this case... but then you have the
> ''interesting'' problem of making the dump driver aware of
that map.
> 
Someone on the ntdev list suggested that one approach could be to
allocate a single page at setup time and map that into each
''hole'' I
leave behind by ballooning pages out. I haven''t checked if Xen supports
a HVM DomU randomly mapping pages like that and if it will let it map
the same page multiple times, but it would remove the errors and keep
everything happy.

James


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>>> "James Harper" <james.harper@bendigoit.com.au>
02.02.10 12:07 >>>
>Someone on the ntdev list suggested that one approach could be to
>allocate a single page at setup time and map that into each
''hole'' I
>leave behind by ballooning pages out. I haven''t checked if Xen
supports
>a HVM DomU randomly mapping pages like that and if it will let it map
>the same page multiple times, but it would remove the errors and keep
>everything happy.
There is no question - it just has to support this. Linux would not work
without this (nor can I imagine any other OS would).

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

James Harper wrote:
> 
> Someone on the ntdev list suggested that one approach could be to
> allocate a single page at setup time and map that into each
''hole'' I
> leave behind by ballooning pages out. I haven''t checked if Xen
supports
> a HVM DomU randomly mapping pages like that and if it will let it map
> the same page multiple times, but it would remove the errors and keep
> everything happy.
> 
Oh that will definitely work but, as I said, the trick is knowing a 
priori where the ''holes'' are :-)

   Paul

-- 
==============================Paul Durrant, Software Engineer

Citrix Systems (R&D) Ltd.
First Floor, Building 101
Cambridge Science Park
Milton Road
Cambridge CB4 0FY
United Kingdom
==============================
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> 
> James Harper wrote:
> >
> > Someone on the ntdev list suggested that one approach could be to
> > allocate a single page at setup time and map that into each
''hole'' I
> > leave behind by ballooning pages out. I haven''t checked if
Xen
supports
> > a HVM DomU randomly mapping pages like that and if it will let it
map
> > the same page multiple times, but it would remove the errors and
keep
> > everything happy.
> >
> 
> Oh that will definitely work but, as I said, the trick is knowing a
> priori where the ''holes'' are :-)
> 
The idea would be that I''d map in my dummy page into each hole as I
make
them, eg at balloon time. It could in theory be done at dump time but
that''s hard, and when a dump occurs you want to do as little as
possible
as the system is in an unknown state. All I do at dump time outside of
my vbd driver is event channel notification and grant ref allocation (my
freelist is lock free now so I don''t have to worry about spinlocks at
DIRQL anymore). Having to look up a table of pages from another driver
altogether would be painful indeed from a scsiport driver.

James


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 02/02/2010 11:27, "Jan Beulich" <JBeulich@novell.com> wrote:
>>>> "James Harper" <james.harper@bendigoit.com.au>
02.02.10 12:07 >>>
>> Someone on the ntdev list suggested that one approach could be to
>> allocate a single page at setup time and map that into each
''hole'' I
>> leave behind by ballooning pages out. I haven''t checked if Xen
supports
>> a HVM DomU randomly mapping pages like that and if it will let it map
>> the same page multiple times, but it would remove the errors and keep
>> everything happy.
> 
> There is no question - it just has to support this. Linux would not work
> without this (nor can I imagine any other OS would).
I don''t think a page can be simultaneously mapped at multiple locations
in a
guest''s p2m though.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

At 13:30 +0000 on 02 Feb (1265117447), Keir Fraser
wrote:
> On 02/02/2010 11:27, "Jan Beulich" <JBeulich@novell.com>
wrote:
> 
> >>>> "James Harper"
<james.harper@bendigoit.com.au> 02.02.10 12:07 >>>
> >> Someone on the ntdev list suggested that one approach could be to
> >> allocate a single page at setup time and map that into each
''hole'' I
> >> leave behind by ballooning pages out. I haven''t checked
if Xen supports
> >> a HVM DomU randomly mapping pages like that and if it will let it
map
> >> the same page multiple times, but it would remove the errors and
keep
> >> everything happy.
> > 
> > There is no question - it just has to support this. Linux would not
work
> > without this (nor can I imagine any other OS would).
> 
> I don''t think a page can be simultaneously mapped at multiple
locations in a
> guest''s p2m though.
No, the basic p2m code doesn''t allow aliasing (because it tries to
maintain a coherent m2p table).  The recent page-sharing patches from
Grzegorz Milos relax that restriction, though IIRC only for explicit
sharing operations managed by dom0.  I''m sure it could be extended to
cover this case.

What''s the problem with just ignoring the errors when a frontend page
doesn''t exist, though?

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>>> Keir Fraser <keir.fraser@eu.citrix.com> 02.02.10 14:30
>>>
>On 02/02/2010 11:27, "Jan Beulich" <JBeulich@novell.com>
wrote:
>
>>>>> "James Harper"
<james.harper@bendigoit.com.au> 02.02.10 12:07 >>>
>>> Someone on the ntdev list suggested that one approach could be to
>>> allocate a single page at setup time and map that into each
''hole'' I
>>> leave behind by ballooning pages out. I haven''t checked if
Xen supports
>>> a HVM DomU randomly mapping pages like that and if it will let it
map
>>> the same page multiple times, but it would remove the errors and
keep
>>> everything happy.
>> 
>> There is no question - it just has to support this. Linux would not
work
>> without this (nor can I imagine any other OS would).
>
>I don''t think a page can be simultaneously mapped at multiple
locations in a
>guest''s p2m though.
Aren''t we talking about HVM guests here? If the guest maps a page, it
doesn''t know about the p2m. Likewise the balloon driver
doesn''t care
about that table (and would install the mapping as James suggested
instead of the [in the Linux balloon driver] pv-guest-only clearing of
the respective 1:1 mapping page table entry). The main problem I
would see him possibly having is that he might need to break up large
pages, but otoh I''m not even sure Windows has a 1:1 mapping.

Confused, Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 02/02/2010 14:26, "Jan Beulich" <JBeulich@novell.com> wrote:
>>> There is no question - it just has to support this. Linux would not
work
>>> without this (nor can I imagine any other OS would).
>> 
>> I don''t think a page can be simultaneously mapped at multiple
locations in a
>> guest''s p2m though.
> 
> Aren''t we talking about HVM guests here? If the guest maps a page,
it
> doesn''t know about the p2m.
I mean that under normal circumstances Xen itself won''t let the page be
installed in multiple p2m slots. It will remove it from the old one beforee
installing it at a new one.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Tim Deegan wrote:
> 
> What''s the problem with just ignoring the errors when a frontend
page
> doesn''t exist, though?
> 
I think the only problem is the noise really. I guess it would be nice 
if we can spot an attempt to grant map a ballooned out page and maybe 
not emit the warning messages in that case.

   Paul

-- 
==============================Paul Durrant, Software Engineer

Citrix Systems (R&D) Ltd.
First Floor, Building 101
Cambridge Science Park
Milton Road
Cambridge CB4 0FY
United Kingdom
==============================
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

At 16:54 +0000 on 02 Feb (1265129651), Paul Durrant
wrote:
> I think the only problem is the noise really. I guess it would be nice 
> if we can spot an attempt to grant map a ballooned out page and maybe 
> not emit the warning messages in that case.
Do we need the warning messages at all?  Can''t we just return an error
to
the caller? 

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Tim Deegan wrote:
> 
> Do we need the warning messages at all?  Can''t we just return an
error to
> the caller? 
> 
As long as we can fail the blkif request in such a way as the frontend 
knows exactly what happened I don''t think there is any need for the
warning.

   Paul

-- 
==============================Paul Durrant, Software Engineer

Citrix Systems (R&D) Ltd.
First Floor, Building 101
Cambridge Science Park
Milton Road
Cambridge CB4 0FY
United Kingdom
==============================
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> >>> Someone on the ntdev list suggested that one approach could be
to
> >>> allocate a single page at setup time and map that into each
''hole'' I
> >>> leave behind by ballooning pages out. I haven''t
checked if Xen supports
> >>> a HVM DomU randomly mapping pages like that and if it will let
it map
> >>> the same page multiple times, but it would remove the errors
and keep
> >>> everything happy.
> >> 
> >> There is no question - it just has to support this. Linux would
not work
> >> without this (nor can I imagine any other OS would).
> >
> >I don''t think a page can be simultaneously mapped at multiple
locations in a
> >guest''s p2m though.
> 
> Aren''t we talking about HVM guests here? If the guest maps a page,
it
> doesn''t know about the p2m. Likewise the balloon driver
doesn''t care
> about that table (and would install the mapping as James suggested
> instead of the [in the Linux balloon driver] pv-guest-only clearing of
> the respective 1:1 mapping page table entry).
The issue is the Windows crash dump driver, which walks over all of
the physical memory which Windows knows about and tries to write it
out.  There''s no way for the PV drivers to tell the crash dump driver
to ignore a particular frame of memory, and so this includes writing
out all the frames which the balloon driver has relinquished.  The
Windows blkfront then ends up sending block requests which reference
ballooned-out memory, which causes an ugly but harmless warning in the
backend.

The obvious fix is for the frontend to just not send those frames, and
to substitute something else in instead, but that means that the
dump-mode block driver needs to be able to get at the list of
ballooned out frames, which is tricky due to architectural issues in
Windows[1].

Alternatively, you could imagine changing the semantics of the balloon
de-populate operation so that it replaces the removed page with, say,
an all-zeroes one, but that then means the P2M is no longer
one-to-one, which is a moderately frightening change just to silence a
warning.  On the other hand, we''re kind of moving in that direction
anyway, with the page sharing stuff and the support for mapping grant
references in HVM guests, so it might be necessary anyway.

Finally, we could just tell James to ignore the warning :), which is
probably the right answer.

[1] Windows crash dump support is quite peculiar.  It''s essentially
kdump-like, except that the kernel which you kexec appears to have
been written from scratch for the specific purpose of writing out
crash dumps and hibernation files, with a compatibility shim so that
it can run normal Windows drivers.  There''s no Microsoft-blessed way
for normal drivers and dump mode drivers to coordinate, and, while
there are a few backdoors, they''re rather unpleasant; we ended up
stashing stuff in xenstore.
> The main problem I would see him possibly having is that he might
> need to break up large pages, but otoh I''m not even sure Windows
has
> a 1:1 mapping.
It doesn''t.

Steven.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> 
> Finally, we could just tell James to ignore the warning :), which is
> probably the right answer.
I''m not _that_ fussed about it, but I do like ''xm
dmesg'' to only tell me
about real problems.

Can Xen tell the difference between a ballooned out page and a missing
page? I suspect it can''t but if it could it would allow an easy way to
turn that message off in that case.
> [1] Windows crash dump support is quite peculiar.  It''s
essentially
> kdump-like, except that the kernel which you kexec appears to have
> been written from scratch for the specific purpose of writing out
> crash dumps and hibernation files, with a compatibility shim so that
> it can run normal Windows drivers.  There''s no Microsoft-blessed
way
> for normal drivers and dump mode drivers to coordinate, and, while
> there are a few backdoors, they''re rather unpleasant; we ended up
> stashing stuff in xenstore.
> 
Reading xenstore at HIGH_LEVEL must be very unpleasant! Do you replicate
the xenstore code in your vbd drivers and poll rather than interrupt at
crash dump time? I map a dummy resources page and store everything
there, which is passed into the crash dump driver so it just picks up
where it left off, and never has to touch xenstore - it just does a very
basic init (picking up the ring pointers etc), cleans up any stale
requests that were submitted before the crash dump and never completed,
then gets to work.

James


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> 
> No, the basic p2m code doesn''t allow aliasing (because it tries to
> maintain a coherent m2p table).  The recent page-sharing patches from
> Grzegorz Milos relax that restriction, though IIRC only for explicit
> sharing operations managed by dom0.  I''m sure it could be extended
to
> cover this case.
> 
> What''s the problem with just ignoring the errors when a frontend
page
> doesn''t exist, though?
> 
It''s mainly the noise. When I''m testing something and I get a
crash, I
like to check ''xm dmesg'' to see if my driver did anything
stupid. During
some recent testing I was mapping an invalid page somewhere (due to
leaking grant refs and not detecting that I''d run out) which gave me a
bunch of errors in ''xm dmesg'', including the ones caused by
the crash
dump itself which confused things a bit.

At the end of the day though, a crash dump only occurs when something
has gone really wrong. Hibernation will have the same problems but there
are only a few cases where you would ever want to hibernate a DomU
anyway.

So if the solution is worse than the problem then I''ve got no objection
to leaving things as they are.

James


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel