llvm dev - May 2012 - [LLVMdev] RFC: R600, a new backend for AMD GPUs

On May 28, 2012 6:44 AM, "Tom Stellard" <thomas.stellard at
amd.com> wrote:
>
> On Fri, May 25, 2012 at 02:37:26PM -0700, Justin Holewinski wrote:
> > Hi Tom,
> >
> > I have a higher-level question regarding this back-end.  If I have an
LLVM
> > IR module and run it through this back-end, it seems like the only
output
> > option is a binary format.  Is this a device binary, or another
> > intermediate format?
> >
> > If the input LLVM IR module was a compute kernel, how would I go about
> > executing it on an AMD GPU?  Can I use the APP SDK to load the binary,
> > perhaps through the CAL interfaces?  How about the OpenCL binary
interface?
> >
>
> Hi Justin,
>
> The binary produced by this backend is meant to be consumed by AMD's
Open
> Source 3D/Compute drivers which are part of the Mesa3D[1] project.  The
> backend is integrated into the driver, so you don't need to compile
> shaders offline.  Currently we are using the backend for graphics and
> compute shaders in our r600g driver (HD2xxx-HD6xxx GPUs) and for graphics
> in our radeonsi (HD7xxx GPUs).  In the future we will use it for compute
> shaders on radensi too.
>
> In order to use the backend for graphics on r600g, you need to build
> Mesa with the --enable-r600-llvm-compiler option.  For compute the
> installation instructions are here:
> http://dri.freedesktop.org/wiki/GalliumCompute
>
> We're working hard to get everything upstream into LLVM to so we can
> have compute shaders working out of the box, so our users don't need to
> manually apply patches.
Okay, so there is no way to use the backend to produce loadable compute
kernels for the proprietary drivers, or on Mac/windows?
>
> Let me know if you have any other questions.
>
> -Tom
>
> [1] http://www.mesa3d.org/
>
> > On Tue, Apr 24, 2012 at 2:52 PM, Tom Stellard <tstellar at
gmail.com>
wrote:
> >
> > > On Mon, Mar 26, 2012 at 12:50:07PM -0400, Tom Stellard wrote:
> > > > Hi,
> > > >
> > > > We've been working on an LLVM backend for the previous
generation
of AMD
> > > > GPUs (HD 2XXX - HD 6XXX) and we would like submit it for
inclusion
in the
> > > > main LLVM tree.  The latest code can be found in this git
repository:
> > > > http://cgit.freedesktop.org/~tstellar/llvm/ in the
r600-initial-review
> > > > branch or if you prefer you can download the entire tree
with this
link:
> > > >
> > >
http://cgit.freedesktop.org/~tstellar/llvm/snapshot/llvm-r600-initial-review.tar.gz
> > > > The R600 backend is located in lib/Target/AMDIL
> > > >
> > > > First, a brief description of the backend:
> > > >
> > > > The r600 backend is being developed as a part of the Open
Source
compute
> > > > stack in Mesa (http://www.mesa3d.org/), which uses the
Gallium API.
> > > > It uses large portions of the AMDIL backend which was
open-sourced
> > > > last December and you'll notice the TargetMachine for
this backend
> > > > (AMDGPUTargetMachine) is a sub-class of AMDILTargetMachine. 
We are
also
> > > > currently working on an LLVM backend for Southern Islands
GPUs, and
we
> > > > would like to get that code into the LLVM tree as well, once
it has
been
> > > > approved for release.  The Southern Islands backend will be
used for
> > > > compute shaders and also graphics shaders in the AMD open
source 3D
> > > driver.
> > > >
> > > > One thing that I would like to point out is that all of the
> > > > code from the AMDIL backend is licensed under a BSD license
with an
> > > > additional clause that deals with United States export laws
(non-AMDIL
> > > > code is licensed with the University of Illinois Open Source
License).
> > > > Will the LLVM project accept a backend with code licensed
under
this BSD
> > > > license?  We would prefer to keep this license, but if it
isn't
> > > > acceptable, we can try to relicense it.
> > > >
> > > > Second, I am looking for two categories of feedback for the
r600
backend:
> > > >
> > > > 1. What changes do I need to make to get the backend
included in the
> > > LLVM tree.
> > > > 2. What changes can I make to improve the backend overall.
> > > >
> > > > My top priority is to get the backend into the LLVM tree, so
> > > > when you provide feedback if you could be clear about what
> > > > changes are needed to get the backend into the tree versus
what
changes
> > > > are just general improvements, I would appreciate it.
> > > >
> > > > Lastly, I did a very brief run through of the code to check
the
coding
> > > style,
> > > > but I know there are still some violations.  For example, a
lot of
the
> > > > file headers are missing file descriptions.  I didn't
want to spend
a
> > > > lot of time on coding style changes prior to the initial
review in
case
> > > > I was asked to make big changes to the code, so I will
address these
> > > > issues once I have received an OK on the organization of the
code.
> > > > However, please still point out coding style errors to me,
and I'll
be
> > > > sure to fix them during the final pass.
> > > >
> > > > Looking forward to your feedback.
> > > >
> > >
> > > Hi,
> > >
> > > Based on some feedback I received on IRC today, it sounds like
this
> > > is just too much code to review at once.  I'm going to try to
strip
> > > the R600 backend down to the bare minimum that is needed for our
Mesa
> > > drivers and then repost the code to the list.  Hopefully, this
will
make it
> > > easier for people to review.
> > >
> > > -Tom
> > > _______________________________________________
> > > LLVM Developers mailing list
> > > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> > > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> > >
> >
> >
> >
> > --
> >
> > Thanks,
> >
> > Justin Holewinski
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20120528/a0475ea0/attachment.html>

On Mon, May 28, 2012 at 08:54:41AM -0700, Justin Holewinski
wrote:
> On May 28, 2012 6:44 AM, "Tom Stellard" <thomas.stellard at
amd.com> wrote:
> >
> > On Fri, May 25, 2012 at 02:37:26PM -0700, Justin Holewinski wrote:
> > > Hi Tom,
> > >
> > > I have a higher-level question regarding this back-end.  If I
have an
> LLVM
> > > IR module and run it through this back-end, it seems like the
only
> output
> > > option is a binary format.  Is this a device binary, or another
> > > intermediate format?
> > >
> > > If the input LLVM IR module was a compute kernel, how would I go
about
> > > executing it on an AMD GPU?  Can I use the APP SDK to load the
binary,
> > > perhaps through the CAL interfaces?  How about the OpenCL binary
> interface?
> > >
> >
> > Hi Justin,
> >
> > The binary produced by this backend is meant to be consumed by
AMD's Open
> > Source 3D/Compute drivers which are part of the Mesa3D[1] project. 
The
> > backend is integrated into the driver, so you don't need to
compile
> > shaders offline.  Currently we are using the backend for graphics and
> > compute shaders in our r600g driver (HD2xxx-HD6xxx GPUs) and for
graphics
> > in our radeonsi (HD7xxx GPUs).  In the future we will use it for
compute
> > shaders on radensi too.
> >
> > In order to use the backend for graphics on r600g, you need to build
> > Mesa with the --enable-r600-llvm-compiler option.  For compute the
> > installation instructions are here:
> > http://dri.freedesktop.org/wiki/GalliumCompute
> >
> > We're working hard to get everything upstream into LLVM to so we
can
> > have compute shaders working out of the box, so our users don't
need to
> > manually apply patches.
> 
> Okay, so there is no way to use the backend to produce loadable compute
> kernels for the proprietary drivers, or on Mac/windows?
>
Right, this is not currently possible.

-Tom
> >
> > Let me know if you have any other questions.
> >
> > -Tom
> >
> > [1] http://www.mesa3d.org/
> >
> > > On Tue, Apr 24, 2012 at 2:52 PM, Tom Stellard <tstellar at
gmail.com>
> wrote:
> > >
> > > > On Mon, Mar 26, 2012 at 12:50:07PM -0400, Tom Stellard
wrote:
> > > > > Hi,
> > > > >
> > > > > We've been working on an LLVM backend for the
previous generation
> of AMD
> > > > > GPUs (HD 2XXX - HD 6XXX) and we would like submit it
for inclusion
> in the
> > > > > main LLVM tree.  The latest code can be found in this
git
> repository:
> > > > > http://cgit.freedesktop.org/~tstellar/llvm/ in the
> r600-initial-review
> > > > > branch or if you prefer you can download the entire
tree with this
> link:
> > > > >
> > > >
>
http://cgit.freedesktop.org/~tstellar/llvm/snapshot/llvm-r600-initial-review.tar.gz
> > > > > The R600 backend is located in lib/Target/AMDIL
> > > > >
> > > > > First, a brief description of the backend:
> > > > >
> > > > > The r600 backend is being developed as a part of the
Open Source
> compute
> > > > > stack in Mesa (http://www.mesa3d.org/), which uses the
Gallium API.
> > > > > It uses large portions of the AMDIL backend which was
open-sourced
> > > > > last December and you'll notice the TargetMachine
for this backend
> > > > > (AMDGPUTargetMachine) is a sub-class of
AMDILTargetMachine.  We are
> also
> > > > > currently working on an LLVM backend for Southern
Islands GPUs, and
> we
> > > > > would like to get that code into the LLVM tree as well,
once it has
> been
> > > > > approved for release.  The Southern Islands backend
will be used for
> > > > > compute shaders and also graphics shaders in the AMD
open source 3D
> > > > driver.
> > > > >
> > > > > One thing that I would like to point out is that all of
the
> > > > > code from the AMDIL backend is licensed under a BSD
license with an
> > > > > additional clause that deals with United States export
laws
> (non-AMDIL
> > > > > code is licensed with the University of Illinois Open
Source
> License).
> > > > > Will the LLVM project accept a backend with code
licensed under
> this BSD
> > > > > license?  We would prefer to keep this license, but if
it isn't
> > > > > acceptable, we can try to relicense it.
> > > > >
> > > > > Second, I am looking for two categories of feedback for
the r600
> backend:
> > > > >
> > > > > 1. What changes do I need to make to get the backend
included in the
> > > > LLVM tree.
> > > > > 2. What changes can I make to improve the backend
overall.
> > > > >
> > > > > My top priority is to get the backend into the LLVM
tree, so
> > > > > when you provide feedback if you could be clear about
what
> > > > > changes are needed to get the backend into the tree
versus what
> changes
> > > > > are just general improvements, I would appreciate it.
> > > > >
> > > > > Lastly, I did a very brief run through of the code to
check the
> coding
> > > > style,
> > > > > but I know there are still some violations.  For
example, a lot of
> the
> > > > > file headers are missing file descriptions.  I
didn't want to spend
> a
> > > > > lot of time on coding style changes prior to the
initial review in
> case
> > > > > I was asked to make big changes to the code, so I will
address these
> > > > > issues once I have received an OK on the organization
of the code.
> > > > > However, please still point out coding style errors to
me, and I'll
> be
> > > > > sure to fix them during the final pass.
> > > > >
> > > > > Looking forward to your feedback.
> > > > >
> > > >
> > > > Hi,
> > > >
> > > > Based on some feedback I received on IRC today, it sounds
like this
> > > > is just too much code to review at once.  I'm going to
try to strip
> > > > the R600 backend down to the bare minimum that is needed for
our Mesa
> > > > drivers and then repost the code to the list.  Hopefully,
this will
> make it
> > > > easier for people to review.
> > > >
> > > > -Tom
> > > > _______________________________________________
> > > > LLVM Developers mailing list
> > > > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> > > > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Thanks,
> > >
> > > Justin Holewinski
> >

On May 28, 2012, at 10:07 AM, Tom Stellard wrote:
>> 
>> Okay, so there is no way to use the backend to produce loadable compute
>> kernels for the proprietary drivers, or on Mac/windows?
>> 
> 
> Right, this is not currently possible.
fwiw..

Sometimes the bleeding edge features found only on Windows (e.g. zero copy) are
interesting to evaluate.
But by in large, HPC folks refuse to even entertain the idea of booting nodes of
a cluster to Windows.
An open source solution like this is very appealing for large-scale scientific
workloads.

Marcus

> -----Original Message-----
> From: Stellard, Thomas
> Sent: Monday, May 28, 2012 9:07 AM
> To: Justin Holewinski
> Cc: Villmow, Micah; Tom Stellard; llvmdev at cs.uiuc.edu
> Subject: Re: [LLVMdev] RFC: R600, a new backend for AMD GPUs
> 
> On Mon, May 28, 2012 at 08:54:41AM -0700, Justin Holewinski wrote:
> > On May 28, 2012 6:44 AM, "Tom Stellard" <thomas.stellard
at amd.com>
> wrote:
> > >
> > > On Fri, May 25, 2012 at 02:37:26PM -0700, Justin Holewinski
wrote:
> > > > Hi Tom,
> > > >
> > > > I have a higher-level question regarding this back-end.  If
I have
> > > > an
> > LLVM
> > > > IR module and run it through this back-end, it seems like
the only
> > output
> > > > option is a binary format.  Is this a device binary, or
another
> > > > intermediate format?
> > > >
> > > > If the input LLVM IR module was a compute kernel, how would
I go
> > > > about executing it on an AMD GPU?  Can I use the APP SDK to
load
> > > > the binary, perhaps through the CAL interfaces?  How about
the
> > > > OpenCL binary
> > interface?
> > > >
> > >
> > > Hi Justin,
> > >
> > > The binary produced by this backend is meant to be consumed by
AMD's
> > > Open Source 3D/Compute drivers which are part of the Mesa3D[1]
> > > project.  The backend is integrated into the driver, so you
don't
> > > need to compile shaders offline.  Currently we are using the
backend
> > > for graphics and compute shaders in our r600g driver
(HD2xxx-HD6xxx
> > > GPUs) and for graphics in our radeonsi (HD7xxx GPUs).  In the
future
> > > we will use it for compute shaders on radensi too.
> > >
> > > In order to use the backend for graphics on r600g, you need to
build
> > > Mesa with the --enable-r600-llvm-compiler option.  For compute
the
> > > installation instructions are here:
> > > http://dri.freedesktop.org/wiki/GalliumCompute
> > >
> > > We're working hard to get everything upstream into LLVM to so
we can
> > > have compute shaders working out of the box, so our users
don't need
> > > to manually apply patches.
> >
> > Okay, so there is no way to use the backend to produce loadable
> > compute kernels for the proprietary drivers, or on Mac/windows?
> >
> 
> Right, this is not currently possible.
[Villmow, Micah] This is not possible with the R600 code generator, yes, but it 
is possible with some ELF fiddling with the AMDIL code generator which the R600
is based on.
> 
> -Tom
> 
> > >
> > > Let me know if you have any other questions.
> > >
> > > -Tom
> > >
> > > [1] http://www.mesa3d.org/
> > >
> > > > On Tue, Apr 24, 2012 at 2:52 PM, Tom Stellard <tstellar
at gmail.com>
> > wrote:
> > > >
> > > > > On Mon, Mar 26, 2012 at 12:50:07PM -0400, Tom Stellard
wrote:
> > > > > > Hi,
> > > > > >
> > > > > > We've been working on an LLVM backend for the
previous
> > > > > > generation
> > of AMD
> > > > > > GPUs (HD 2XXX - HD 6XXX) and we would like submit
it for
> > > > > > inclusion
> > in the
> > > > > > main LLVM tree.  The latest code can be found in
this git
> > repository:
> > > > > > http://cgit.freedesktop.org/~tstellar/llvm/ in the
> > r600-initial-review
> > > > > > branch or if you prefer you can download the
entire tree with
> > > > > > this
> > link:
> > > > > >
> > > > >
> > http://cgit.freedesktop.org/~tstellar/llvm/snapshot/llvm-r600-initial-
> > review.tar.gz
> > > > > > The R600 backend is located in lib/Target/AMDIL
> > > > > >
> > > > > > First, a brief description of the backend:
> > > > > >
> > > > > > The r600 backend is being developed as a part of
the Open
> > > > > > Source
> > compute
> > > > > > stack in Mesa (http://www.mesa3d.org/), which uses
the Gallium
> API.
> > > > > > It uses large portions of the AMDIL backend which
was
> > > > > > open-sourced last December and you'll notice
the TargetMachine
> > > > > > for this backend
> > > > > > (AMDGPUTargetMachine) is a sub-class of
AMDILTargetMachine.
> > > > > > We are
> > also
> > > > > > currently working on an LLVM backend for Southern
Islands
> > > > > > GPUs, and
> > we
> > > > > > would like to get that code into the LLVM tree as
well, once
> > > > > > it has
> > been
> > > > > > approved for release.  The Southern Islands
backend will be
> > > > > > used for compute shaders and also graphics shaders
in the AMD
> > > > > > open source 3D
> > > > > driver.
> > > > > >
> > > > > > One thing that I would like to point out is that
all of the
> > > > > > code from the AMDIL backend is licensed under a
BSD license
> > > > > > with an additional clause that deals with United
States export
> > > > > > laws
> > (non-AMDIL
> > > > > > code is licensed with the University of Illinois
Open Source
> > License).
> > > > > > Will the LLVM project accept a backend with code
licensed
> > > > > > under
> > this BSD
> > > > > > license?  We would prefer to keep this license,
but if it
> > > > > > isn't acceptable, we can try to relicense it.
> > > > > >
> > > > > > Second, I am looking for two categories of
feedback for the
> > > > > > r600
> > backend:
> > > > > >
> > > > > > 1. What changes do I need to make to get the
backend included
> > > > > > in the
> > > > > LLVM tree.
> > > > > > 2. What changes can I make to improve the backend
overall.
> > > > > >
> > > > > > My top priority is to get the backend into the
LLVM tree, so
> > > > > > when you provide feedback if you could be clear
about what
> > > > > > changes are needed to get the backend into the
tree versus
> > > > > > what
> > changes
> > > > > > are just general improvements, I would appreciate
it.
> > > > > >
> > > > > > Lastly, I did a very brief run through of the code
to check
> > > > > > the
> > coding
> > > > > style,
> > > > > > but I know there are still some violations.  For
example, a
> > > > > > lot of
> > the
> > > > > > file headers are missing file descriptions.  I
didn't want to
> > > > > > spend
> > a
> > > > > > lot of time on coding style changes prior to the
initial
> > > > > > review in
> > case
> > > > > > I was asked to make big changes to the code, so I
will address
> > > > > > these issues once I have received an OK on the
organization of
> the code.
> > > > > > However, please still point out coding style
errors to me, and
> > > > > > I'll
> > be
> > > > > > sure to fix them during the final pass.
> > > > > >
> > > > > > Looking forward to your feedback.
> > > > > >
> > > > >
> > > > > Hi,
> > > > >
> > > > > Based on some feedback I received on IRC today, it
sounds like
> > > > > this is just too much code to review at once.  I'm
going to try
> > > > > to strip the R600 backend down to the bare minimum that
is
> > > > > needed for our Mesa drivers and then repost the code to
the
> > > > > list.  Hopefully, this will
> > make it
> > > > > easier for people to review.
> > > > >
> > > > > -Tom
> > > > > _______________________________________________
> > > > > LLVM Developers mailing list
> > > > > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> > > > > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Thanks,
> > > >
> > > > Justin Holewinski
> > >