Xen devel - Nov 2008 - Infinte loop in RtlPrefetchMemoryNonTemporal under Windows

I have been trying to track down a problem that occurs when my GPLPV
drivers give Windows a packet with the NdisPacketTcpChecksumSucceeded
flag set. The problem is that the machine locks hard.

After some tedious work with the debugger, I have found that Windows
makes a call to a routine called RtlPrefetchMemoryNonTemporal, which
looks like this:

8088e579 mov     eax,dword ptr [nt!KePrefetchNTAGranularity 8088e57e
0f184100 prefetchnta [ecx]
8088e582 add     ecx,eax
8088e584 sub     edx,eax
8088e586 ja      nt!RtlPrefetchMemoryNonTemporal+0x6 (8088e57e)
8088e588 ret

Unfortunately it appears that the value at nt!KePrefetchNTAGranularity
is 0, hence the infinite loop.

Is there anything in Xen that could cause Windows to put a zero in
there? The other (more likely?) alternative is that I''m doing something
wrong in my driver that is causing that value to be stepped on. It seems
really strange that I could so neatly step on that value without
breaking the machine in countless other ways though...

The problem only appeared when I upgraded from 3.2.1 to 3.3.0 too.

Any suggestions? It''s driving me crazy!

Thanks

James

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> I have been trying to track down a problem that occurs when my GPLPV
> drivers give Windows a packet with the NdisPacketTcpChecksumSucceeded
> flag set. The problem is that the machine locks hard.
> 
> After some tedious work with the debugger, I have found that Windows
> makes a call to a routine called RtlPrefetchMemoryNonTemporal, which
> looks like this:
> 
> 8088e579 mov     eax,dword ptr [nt!KePrefetchNTAGranularity 8088e57e
> 0f184100 prefetchnta [ecx]
> 8088e582 add     ecx,eax
> 8088e584 sub     edx,eax
> 8088e586 ja      nt!RtlPrefetchMemoryNonTemporal+0x6 (8088e57e)
> 8088e588 ret
> 
> Unfortunately it appears that the value at nt!KePrefetchNTAGranularity
> is 0, hence the infinite loop.
> 
A bit more debugging shows that RtlPrefectMemoryNonTemporal starts out
life like this:

nt!RtlPrefetchMemoryNonTemporal:
8088e578 ret
8088e579 mov     eax,dword ptr [nt!KePrefetchNTAGranularity 8088e57e
0f184100 prefetchnta [ecx]
8088e582 add     ecx,eax
8088e584 sub     edx,eax
8088e586 ja      nt!RtlPrefetchMemoryNonTemporal+0x6 (8088e57e)
8088e588 ret

But then for some reason becomes:

nt!RtlPrefetchMemoryNonTemporal:
8088e578 nop
8088e579 mov     eax,dword ptr [nt!KePrefetchNTAGranularity 8088e57e
0f184100 prefetchnta [ecx]
8088e582 add     ecx,eax
8088e584 sub     edx,eax
8088e586 ja      nt!RtlPrefetchMemoryNonTemporal+0x6 (8088e57e)
8088e588 ret

The difference being that the initial ret becomes a nop...

This is getting stranger and stranger.

James

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I fixed this with the following entry in my config:

cpuid
[''0x80000005:ecx=xxxxxxxxxxxxxxxxxxxxxxxxkkkkkkkk,edx=xxxxxxxxxxxxxxxxxx
xxxxxxkkkkkkkk'']

This passes through the physical host value of the relevant cache
information cpuid function to the DomU.

Is there any reason why we wouldn''t want to do this? I think
I''m working
around what is actually a bug. Intel CPU''s seem to keep cache
information in another cpuid function so maybe it is only (some?) AMD
cpu''s affected?

As per the email below, under some circumstances (eg when I give Windows
a pre-checksum-verified packet) it makes a call to a function called
RtlPrefetchMemoryNonTemporal which uses the cache line size as a
decrement in a loop. Xen is setting the cache line size to zero (at
least for AMD processors) causing that loop to never terminate.

James
> -----Original Message-----
> From: James Harper
> Sent: Saturday, 22 November 2008 23:00
> To: James Harper; xen-devel@lists.xensource.com
> Subject: RE: [Xen-devel] Infinte loop in RtlPrefetchMemoryNonTemporal
> underWindows
> 
> > I have been trying to track down a problem that occurs when my GPLPV
> > drivers give Windows a packet with the
NdisPacketTcpChecksumSucceeded
> > flag set. The problem is that the machine locks hard.
> >
> > After some tedious work with the debugger, I have found that Windows
> > makes a call to a routine called RtlPrefetchMemoryNonTemporal, which
> > looks like this:
> >
> > 8088e579 mov     eax,dword ptr [nt!KePrefetchNTAGranularity 8088e57e
> > 0f184100 prefetchnta [ecx]
> > 8088e582 add     ecx,eax
> > 8088e584 sub     edx,eax
> > 8088e586 ja      nt!RtlPrefetchMemoryNonTemporal+0x6 (8088e57e)
> > 8088e588 ret
> >
> > Unfortunately it appears that the value at
nt!KePrefetchNTAGranularity
> > is 0, hence the infinite loop.
> >
> 
> A bit more debugging shows that RtlPrefectMemoryNonTemporal starts out
> life like this:
> 
> nt!RtlPrefetchMemoryNonTemporal:
> 8088e578 ret
> 8088e579 mov     eax,dword ptr [nt!KePrefetchNTAGranularity 8088e57e
> 0f184100 prefetchnta [ecx]
> 8088e582 add     ecx,eax
> 8088e584 sub     edx,eax
> 8088e586 ja      nt!RtlPrefetchMemoryNonTemporal+0x6 (8088e57e)
> 8088e588 ret
> 
> But then for some reason becomes:
> 
> nt!RtlPrefetchMemoryNonTemporal:
> 8088e578 nop
> 8088e579 mov     eax,dword ptr [nt!KePrefetchNTAGranularity 8088e57e
> 0f184100 prefetchnta [ecx]
> 8088e582 add     ecx,eax
> 8088e584 sub     edx,eax
> 8088e586 ja      nt!RtlPrefetchMemoryNonTemporal+0x6 (8088e57e)
> 8088e588 ret
> 
> The difference being that the initial ret becomes a nop...
> 
> This is getting stranger and stranger.
> 
> James
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

James,

That should be fixed be the changeset 18512.
Does your libxc is up to date ?

2008/11/22 James Harper
<james.harper@bendigoit.com.au>:
> I fixed this with the following entry in my config:
>
> cpuid >
[''0x80000005:ecx=xxxxxxxxxxxxxxxxxxxxxxxxkkkkkkkk,edx=xxxxxxxxxxxxxxxxxx
> xxxxxxkkkkkkkk'']
>
> This passes through the physical host value of the relevant cache
> information cpuid function to the DomU.
>
> Is there any reason why we wouldn''t want to do this? I think
I''m working
> around what is actually a bug. Intel CPU''s seem to keep cache
> information in another cpuid function so maybe it is only (some?) AMD
> cpu''s affected?
>
> As per the email below, under some circumstances (eg when I give Windows
> a pre-checksum-verified packet) it makes a call to a function called
> RtlPrefetchMemoryNonTemporal which uses the cache line size as a
> decrement in a loop. Xen is setting the cache line size to zero (at
> least for AMD processors) causing that loop to never terminate.
>
> James
>
>> -----Original Message-----
>> From: James Harper
>> Sent: Saturday, 22 November 2008 23:00
>> To: James Harper; xen-devel@lists.xensource.com
>> Subject: RE: [Xen-devel] Infinte loop in RtlPrefetchMemoryNonTemporal
>> underWindows
>>
>> > I have been trying to track down a problem that occurs when my
GPLPV
>> > drivers give Windows a packet with the
> NdisPacketTcpChecksumSucceeded
>> > flag set. The problem is that the machine locks hard.
>> >
>> > After some tedious work with the debugger, I have found that
Windows
>> > makes a call to a routine called RtlPrefetchMemoryNonTemporal,
which
>> > looks like this:
>> >
>> > 8088e579 mov     eax,dword ptr [nt!KePrefetchNTAGranularity
8088e57e
>> > 0f184100 prefetchnta [ecx]
>> > 8088e582 add     ecx,eax
>> > 8088e584 sub     edx,eax
>> > 8088e586 ja      nt!RtlPrefetchMemoryNonTemporal+0x6 (8088e57e)
>> > 8088e588 ret
>> >
>> > Unfortunately it appears that the value at
> nt!KePrefetchNTAGranularity
>> > is 0, hence the infinite loop.
>> >
>>
>> A bit more debugging shows that RtlPrefectMemoryNonTemporal starts out
>> life like this:
>>
>> nt!RtlPrefetchMemoryNonTemporal:
>> 8088e578 ret
>> 8088e579 mov     eax,dword ptr [nt!KePrefetchNTAGranularity 8088e57e
>> 0f184100 prefetchnta [ecx]
>> 8088e582 add     ecx,eax
>> 8088e584 sub     edx,eax
>> 8088e586 ja      nt!RtlPrefetchMemoryNonTemporal+0x6 (8088e57e)
>> 8088e588 ret
>>
>> But then for some reason becomes:
>>
>> nt!RtlPrefetchMemoryNonTemporal:
>> 8088e578 nop
>> 8088e579 mov     eax,dword ptr [nt!KePrefetchNTAGranularity 8088e57e
>> 0f184100 prefetchnta [ecx]
>> 8088e582 add     ecx,eax
>> 8088e584 sub     edx,eax
>> 8088e586 ja      nt!RtlPrefetchMemoryNonTemporal+0x6 (8088e57e)
>> 8088e588 ret
>>
>> The difference being that the initial ret becomes a nop...
>>
>> This is getting stranger and stranger.
>>
>> James
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>


-- 
Jean Guyader

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> James,
> 
> That should be fixed be the changeset 18512.
> Does your libxc is up to date ?
> 
Obviously that''s not in 3.3.0... is it going to be part of the next
release?

Thanks

James

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 22/11/08 21:42, "James Harper"
<james.harper@bendigoit.com.au> wrote:
>> James,
>> 
>> That should be fixed be the changeset 18512.
>> Does your libxc is up to date ?
>> 
> 
> Obviously that''s not in 3.3.0... is it going to be part of the
next
> release?
Yes, it''s in xen-3.3-testing.hg already and will be in 3.3.1.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel