Nouveau - Oct 2019 - [PATCH RFC v4 14/16] drm, cgroup: Introduce lgpu as DRM cgroup resource

On Wed, Oct 9, 2019 at 8:52 PM Greathouse, Joseph
<Joseph.Greathouse at amd.com> wrote:
>
> > From: Daniel Vetter <daniel.vetter at ffwll.ch> On Behalf Of
Daniel Vetter
> > Sent: Wednesday, October 9, 2019 11:07 AM
> > On Wed, Oct 09, 2019 at 03:53:42PM +0000, Kuehling, Felix wrote:
> > > On 2019-10-09 11:34, Daniel Vetter wrote:
> > > > On Wed, Oct 09, 2019 at 03:25:22PM +0000, Kuehling, Felix
wrote:
> > > >> On 2019-10-09 6:31, Daniel Vetter wrote:
> > > >>> On Tue, Oct 08, 2019 at 06:53:18PM +0000, Kuehling,
Felix wrote:
> > > >>>> The description sounds reasonable to me and maps
well to the CU masking
> > > >>>> feature in our GPUs.
> > > >>>>
> > > >>>> It would also allow us to do more coarse-grained
masking for example to
> > > >>>> guarantee balanced allocation of CUs across
shader engines or
> > > >>>> partitioning of memory bandwidth or CP pipes (if
that is supported by
> > > >>>> the hardware/firmware).
> > > >>> Hm, so this sounds like the definition for how this
cgroup is supposed to
> > > >>> work is "amd CU masking" (whatever that
exactly is). And the abstract
> > > >>> description is just prettification on top, but not
actually the real
> > > >>> definition you guys want.
> > > >> I think you're reading this as the opposite of what
I was trying to say.
> > > >> Using CU masking is one possible implementation of LGPUs
on AMD
> > > >> hardware. It's the one that Kenny implemented at the
end of this patch
> > > >> series, and I pointed out some problems with that
approach. Other ways
> > > >> to partition the hardware into LGPUs are conceivable.
For example we're
> > > >> considering splitting it along the lines of shader
engines, which is
> > > >> more coarse-grain and would also affect memory bandwidth
available to
> > > >> each partition.
> > > > If this is supposed to be useful for admins then "other
ways to partition
> > > > the hw are conceivable" is the problem. This should be
unique&clear for
> > > > admins/end-users. Reading the implementation details and
realizing that
> > > > the actual meaning is "amd CU masking" isn't
good enough by far, since
> > > > that's meaningless on any other hw.
> > > >
> > > > And if there's other ways to implement this cgroup for
amd, it's also
> > > > meaningless (to sysadmins/users) for amd hw.
> > > >
> > > >> We could also consider partitioning pipes in our command
processor,
> > > >> although that is not supported by our current CP
scheduler firmware.
> > > >>
> > > >> The bottom line is, the LGPU model proposed by Kenny is
quite abstract
> > > >> and allows drivers implementing it a lot of flexibility
depending on the
> > > >> capability of their hardware and firmware. We
haven't settled on a final
> > > >> implementation choice even for AMD.
> > > > That abstract model of essentially "anything goes"
is the problem here
> > > > imo. E.g. for cpu cgroups this would be similar to allowing
the bitmaks to
> > > > mean "cpu core" on one machine "physical
die" on the next and maybe
> > > > "hyperthread unit" on the 3rd. Useless for admins.
> > > >
> > > > So if we have a gpu bitmaks thing that might mean a command
submissio pipe
> > > > on one hw (maybe matching what vk exposed, maybe not), some
compute unit
> > > > mask on the next and something entirely different (e.g.
intel has so
> > > > called GT slices with compute cores + more stuff around) on
the 3rd vendor
> > > > then that's not useful for admins.
> > >
> > > The goal is to partition GPU compute resources to eliminate as
much
> > > resource contention as possible between different partitions.
Different
> > > hardware will have different capabilities to implement this. No
> > > implementation will be perfect. For example, even with CPU cores
that
> > > are supposedly well defined, you can still have different
behaviours
> > > depending on CPU cache architectures, NUMA and thermal management
across
> > > CPU cores. The admin will need some knowledge of their hardware
> > > architecture to understand those effects that are not described
by the
> > > abstract model of cgroups.
> >
> > That's not the point I was making. For cpu cgroups there's a
very well
> > defined connection between the cpu bitmasks/numbers in cgroups and the
cpu
> > bitmasks you use in various system calls (they match). And that stuff
> > works across vendors.
> >
> > We need the same for gpus.
> >
> > > The LGPU model is deliberately flexible, because GPU
architectures are
> > > much less standardized than CPU architectures. Expecting a common
model
> > > that is both very specific and applicable to to all GPUs is
unrealistic,
> > > in my opinion.
> >
> > So pure abstraction isn't useful, we need to know what these bits
mean.
> > Since if they e.g. mean vk pipes, then maybe I shouldn't be using
those vk
> > pipes in my application anymore. Or we need to define that the
userspace
> > driver needs to filter out any pipes that arent' accessible (if
that's
> > possible, no idea).
> >
> > cgroups that essentially have pure hw depedent meaning aren't
useful.
> > Note: this is about the fundamental meaning, not about the more
unclear
> > isolation guarantees (which are indeed hw specific on different cpu
> > platforms). We're not talking about "different gpus might
have different
> > amounts of shared caches bitween different bitmasks". We're
talking
> > "different gpus might assign completely differen meaning to these
> > bitmasks".
> > -Daniel
> <snip>
>
> One thing that comes to mind is the OpenCL 1.2+ SubDevices mechanism:
https://www.khronos.org/registry/OpenCL/sdk/1.2/docs/man/xhtml/clCreateSubDevices.html
>
> The concept of LGPU in cgroups seems to match up nicely with an OpenCL
SubDevice, at least for compute tasks. We want to divide up the device and give
some configurable subset of it to the user as a logical GPU or sub-device.
>
> OpenCL defines Compute Units (CUs), and any GPU vendor that runs OpenCL has
some mapping of their internal compute resources to this concept of CUs. Off the
top of my head (I may be misremembering some of these):
> - AMD: Compute Units (CUs)
> - ARM: Shader Cores (SCs)
> - Intel: Execution Units (EUs)
> - Nvidia: Streaming Multiprocessors (SMs)
> - Qualcomm: Shader Processors (SPs)
>
> The clCreateSubDevices() API has a variety of ways to slice and dice these
compute resources across sub-devices. PARTITION_EQUALLY and PARTITION_BY_COUNTS
could possibly be handled by a simple high-level mechanism that just allows you
to request some percentage of the available GPU compute resources.
>
> PARTITION_BY_AFFINITY_DOMAIN, however, splits up the CUs based on
lower-level information such as what cache levels are shared or what NUMA domain
a collection of CUs is in. I would argue that a runtime that wants to do this
needs to know a bit about the mapping of CUs to underlying hardware resources.
>
> A cgroup implementation that presented a CU bitmap could sit at the bottom
of all three of these partitioning schemes, and more advanced ones if they come
up. We might be getting side-tracked by the fact that AMD calls its resources
CUs. The OpenCL (or Vulkan, etc.) concept of a Compute Unit is cross-vendor. The
concept of targeting work to [Khronos-defined] Compute Units isn't
AMD-specific. A bitmap of [Khronos-defined] CUs could map to any of these broad
vendor compute resources.
>
> There may be other parts of the GPU that we want to divide up -- command
queue resources, pipes, render backends, etc. I'm not sure if any of those
have been "standardized" between GPUs to such an extent that they make
sense to put into cgroups yet -- I'm ignorant outside of the compute world.
But at least the concept of CUs (or SMs, or EUs, etc.) seems to be standard
across GPUs and (to me anyway) seems like a reasonable place to allows
administrators, developers, users, etc. to divide up their GPUs.
>
> And whatever mechanisms a GPU vendor may put in place to do
clCreateSubDevices() could then be additionally used inside the kernel for their
cgroups LGPU partitioning.
Yeah this is the stuff I meant. I quickly checked intel's CL driver,
and from a quick look we don't support that. Adding Karol, who might
know whether this works on nvidia hw and how. If opencl CU don't
really apply to more than amdgpu, then that's not really helping much
with making this stuff more broadly useful.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch