Nouveau - Nov 2015 - Documentation request for MP warp error 0x10

On Fri, Oct 02, 2015 at 06:05:21PM -0400, Ilia Mirkin
wrote:
> Could you advise what the proper way of indicating
> that the memory is "global" to the op? I'm sure I'm just
missing
> something simple. If you show me what to look for in SM35 I can
> probably find it on my own for SM20/SM30/SM50.
Sorry again for the delay.  Here's what I've been able to find out about
the generic thread address space (used by the SMs) and what types of
memory it contains.  Hopefully this clears things up.


Local memory is a per-thread space.
Shared memory is a per-CTA space (compute shaders only).

LDL and STL instructions access local memory with a zero offset.
LDS, LSDLK, STS, and STSCUL instructions access shared memory with a zero
offset.

LD, ST, RED, ATOM, and CCTL.D instructions access the generic thread address
space, which is layered on top of the channel's virtual address space.

In the generic thread address space, there are 16MB windows into local and
shared memory; everything not in a Local or Shared address window accesses
global virtual memory.

The local window offset within the generic thread address space is determined
by the SetShaderLocalMemoryWindow class method (offset 0x77c in classes *97 and
*c0).

The shared window offset within the generic thread address space is determined
by the SetShaderSharedMemoryWindow class method (offset 0x214 in classes *c0).

For both methods, the offset is in bytes, but the window must be aligned to a
16MB boundary (so the lower 24 bits of the data must be zero). The upper 32
bits of the windows are hard-coded to 0 (so they must be placed within the
lower 4GB of address space).

Generally, it is expected that software will reserve ranges in the global
virtual address space where these windows will be placed. (Otherwise anything
mapped there will be inaccessible to shaders.)

For graphics shaders, the shared address space logic does not exist, so there
is no need to reserve virtual memory for it.


- Robert

On Fri, Nov 6, 2015 at 3:59 PM, Robert Morell <rmorell at nvidia.com>
wrote:
> On Fri, Oct 02, 2015 at 06:05:21PM -0400, Ilia Mirkin wrote:
>> Could you advise what the proper way of indicating
>> that the memory is "global" to the op? I'm sure I'm
just missing
>> something simple. If you show me what to look for in SM35 I can
>> probably find it on my own for SM20/SM30/SM50.
>
> Sorry again for the delay.  Here's what I've been able to find out
about
> the generic thread address space (used by the SMs) and what types of
> memory it contains.  Hopefully this clears things up.
>
>
> Local memory is a per-thread space.
> Shared memory is a per-CTA space (compute shaders only).
>
> LDL and STL instructions access local memory with a zero offset.
> LDS, LSDLK, STS, and STSCUL instructions access shared memory with a zero
> offset.
>
> LD, ST, RED, ATOM, and CCTL.D instructions access the generic thread
address
> space, which is layered on top of the channel's virtual address space.
>
> In the generic thread address space, there are 16MB windows into local and
> shared memory; everything not in a Local or Shared address window accesses
> global virtual memory.
>
> The local window offset within the generic thread address space is
determined
> by the SetShaderLocalMemoryWindow class method (offset 0x77c in classes *97
and
> *c0).
>
> The shared window offset within the generic thread address space is
determined
> by the SetShaderSharedMemoryWindow class method (offset 0x214 in classes
*c0).
>
> For both methods, the offset is in bytes, but the window must be aligned to
a
> 16MB boundary (so the lower 24 bits of the data must be zero). The upper 32
> bits of the windows are hard-coded to 0 (so they must be placed within the
> lower 4GB of address space).
>
> Generally, it is expected that software will reserve ranges in the global
> virtual address space where these windows will be placed. (Otherwise
anything
> mapped there will be inaccessible to shaders.)
>
> For graphics shaders, the shared address space logic does not exist, so
there
> is no need to reserve virtual memory for it.
Hi Robert, thanks so much for getting back to me. I believe I've
understood what you've said, but please confirm:

In order for ATOM.*/RED.* to work, the addresses in question must
*NOT* be inside of the 16MB local/shared windows. So if I'm getting
that error, the address must be inside.

If so, this may be a reasonable explanation for what I'm seeing --
while I knew about the local/shared windows, I didn't realize that the
windows were 16MB-sized.

Thanks again,

  -ilia

On Fri, Nov 06, 2015 at 04:15:29PM -0500, Ilia Mirkin
wrote:
> In order for ATOM.*/RED.* to work, the addresses in question must
> *NOT* be inside of the 16MB local/shared windows. So if I'm getting
> that error, the address must be inside.
Yes, that's my understanding.
> If so, this may be a reasonable explanation for what I'm seeing --
Cool, I'm happy it helps.


Thanks,
Robert