Hi, for a few months already I have been asking around for opinions on how people could best work together on Khronos' SPIR-V <-> LLVM-IR converter and some consensus seems to have formed. Most of the people I talked to favored having the converter become an external tool within the LLVM project, similar to libclc. I think that the LLVM project's processes, infrastructure and community form the best place for this collaboration to happen. I hope that having the converter as part of LLVM can help expand LLVM's value proposition in heterogeneous computing, in part by complementing the OpenCL C backend in clang. Thus I would like to ask what the LLVM community needs to see from us before SPIRV-LLVM can be accepted as an external tool, if at all. Thanks, Tomeu
Nicholas Wilson via llvm-dev
2018-Feb-21 09:58 UTC
[llvm-dev] SPIRV-LLVM as an external tool
I agree that having the conversion live as part of LLVM is the best option going forward. libclc is an library implementation of the OpenCL builtins and I think its fine for that aspect of the converter, i.e. consumption of SPIR-V into LLVM IR in preparation for code generation by LLVM by its various backends as a library to link the generated code to, to live in an external repo. This is fine because SPIR<->LLVM IR is trivial, SPIR-V<->LLVM is not. Having the SPIR-V <-> LLVM-IR conversion part of the tool as an external repo as opposed to the llvm would make it very difficult to use as a fontend: First the changes to the build pipeline are a hinderance. In my fork of SPIRV-LLVM the backend is a target because I couldn’t get CMake to link the libraries to LDC as a support library. This would be made even more complicated with at external project. I would probably resort to invoking an external program despite integrated conversion being the best solution. Secondly (unless you've changed things) all of the builtins are not intrinsics, but C++ Itanium with extensions mangled. This makes it so that anyone not implementing a C++ frontend has to deal with it, presumably by hardcoding the names since C++ is very non-trivial. Even for projects like LDC (the LLVM based D compiler) which have a C++ mangler, we still have to hardcode the names because Windows does not use Itanium and the compiler is not set up to do two different forms of C++ mangling, and even if that were possible the non-standard extensions make it infeasible. Whereas any LLVM frontend worth is salt has a way of exposing intrinsics. The backend as it currently stands is not based on tablegen and that is required AFAIU to support intrinsics. I have tablegen descriptions of the SPIR-V format mostly complete (all core instruction relevant to OpenCL + most of the ones for Vulkan, + all the OpenCL extension instructions) though not the intrinsics. However I didn’t get around to using them because I have more important things to work on in the front end. Then there is the question of what to do about Vulkan? I would love to have support for Vulkan in LLVM. That would be relatively simple to set up for with SPIRV as a target, even more so with a tablegen based infrastructure. I have no idea how that would work as an external project. I also think that having it in an external project will greatly reduce the visibility in comparison to a target. How many of you have cloned libclc for example? Not many I suspect. Probably because you are not working on implementing OpenCL for a backend. And while libclc is useful to those implementing OpenCL they are vastly outnumbered by those using OpenCL i.e. producing SPIR/SPIR-V. My usage of SPIR-V is I generate it from my source files though LDC/LLVM then I hand it off to the OpenCL implementation. Nowhere in that pipeline do I need something like libclc. Now libclc may be used in the implementation but that’s a different copy of LLVM to the one that produced it. TL;DR. This really needs to be a target so that we can use intrinsics, have it as part of regular LLVM releases so that it can be used by many frontends. Having an external project for assisting backends consuming SPIR-V (a la libclc) is fine but that shouldn’t mean that the processes used for generating SPIR-V need to go into a separate project. Nic> On 21 Feb 2018, at 4:15 pm, Tomeu Vizoso via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > Hi, > > for a few months already I have been asking around for opinions on how > people could best work together on Khronos' SPIR-V <-> LLVM-IR converter > and some consensus seems to have formed. > > Most of the people I talked to favored having the converter become an > external tool within the LLVM project, similar to libclc. I think that > the LLVM project's processes, infrastructure and community form the best > place for this collaboration to happen. > > I hope that having the converter as part of LLVM can help expand LLVM's > value proposition in heterogeneous computing, in part by complementing > the OpenCL C backend in clang. > > Thus I would like to ask what the LLVM community needs to see from us > before SPIRV-LLVM can be accepted as an external tool, if at all. > > Thanks, > > Tomeu > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
On 02/21/2018 10:58 AM, Nicholas Wilson wrote:> I agree that having the conversion live as part of LLVM is the best option going forward. > > libclc is an library implementation of the OpenCL builtins and I think its fine for that aspect of the converter, > i.e. consumption of SPIR-V into LLVM IR in preparation for code generation by LLVM by its various backends as a library > to link the generated code to, to live in an external repo. > > This is fine because SPIR<->LLVM IR is trivial, SPIR-V<->LLVM is not. > > Having the SPIR-V <-> LLVM-IR conversion part of the tool as an external repo as opposed to the llvm would > make it very difficult to use as a fontend: > > First the changes to the build pipeline are a hinderance. In my fork of SPIRV-LLVM the backend is a target > because I couldn’t get CMake to link the libraries to LDC as a support library. This would be made even more > complicated with at external project. I would probably resort to invoking an external program despite integrated > conversion being the best solution. > > Secondly (unless you've changed things) all of the builtins are not intrinsics, but C++ Itanium with extensions mangled. > This makes it so that anyone not implementing a C++ frontend has to deal with it, presumably by hardcoding the > names since C++ is very non-trivial. Even for projects like LDC (the LLVM based D compiler) which have a C++ mangler, > we still have to hardcode the names because Windows does not use Itanium and the compiler is not set up to do two > different forms of C++ mangling, and even if that were possible the non-standard extensions make it infeasible. > Whereas any LLVM frontend worth is salt has a way of exposing intrinsics. > > The backend as it currently stands is not based on tablegen and that is required AFAIU to support intrinsics. > I have tablegen descriptions of the SPIR-V format mostly complete (all core instruction relevant to OpenCL + > most of the ones for Vulkan, + all the OpenCL extension instructions) though not the intrinsics. > However I didn’t get around to using them because I have more important things to work on in the front end. > > Then there is the question of what to do about Vulkan? I would love to have support for Vulkan in LLVM. > That would be relatively simple to set up for with SPIRV as a target, even more so with a tablegen based > infrastructure. I have no idea how that would work as an external project. > > I also think that having it in an external project will greatly reduce the visibility in comparison to a target. > How many of you have cloned libclc for example? Not many I suspect. Probably because you are not > working on implementing OpenCL for a backend. And while libclc is useful to those implementing OpenCL > they are vastly outnumbered by those using OpenCL i.e. producing SPIR/SPIR-V. My usage of SPIR-V is > I generate it from my source files though LDC/LLVM then I hand it off to the OpenCL implementation. > Nowhere in that pipeline do I need something like libclc. Now libclc may be used in the implementation > but that’s a different copy of LLVM to the one that produced it. > > TL;DR. > > This really needs to be a target so that we can use intrinsics, have it as part of regular LLVM releases > so that it can be used by many frontends. Having an external project for assisting backends consuming > SPIR-V (a la libclc) is fine but that shouldn’t mean that the processes used for generating SPIR-V need to go > into a separate project.Yes, for compilers it would be much more convenient to have a backend, but what OpenCL implementations need (for example) is only a bidirectional converter. I don't think anybody will oppose to the converter becoming a backend at some point (as long as we keep exposing the conversion passes as today), but at the current pace it will take years and I don't think it's fair for everybody that we keep this codebase in the limbo for much longer. Currently there's people shipping this code in products, and have to carry heavily modified forks because there's no community upstream to contribute to. So, I think we really need at this point to find some place where the community can gather and work together on stability, intrinsics, have a proper backend, etc. This proposal is only about where the collaboration would happen in the near future. If at some point we have also a backend, and people want to have it along the other backends in the main LLVM repo, that will be great. But in the meantime we need a place to work together. Regards, Tomeu> Nic > >> On 21 Feb 2018, at 4:15 pm, Tomeu Vizoso via llvm-dev <llvm-dev at lists.llvm.org> wrote: >> >> Hi, >> >> for a few months already I have been asking around for opinions on how >> people could best work together on Khronos' SPIR-V <-> LLVM-IR converter >> and some consensus seems to have formed. >> >> Most of the people I talked to favored having the converter become an >> external tool within the LLVM project, similar to libclc. I think that >> the LLVM project's processes, infrastructure and community form the best >> place for this collaboration to happen. >> >> I hope that having the converter as part of LLVM can help expand LLVM's >> value proposition in heterogeneous computing, in part by complementing >> the OpenCL C backend in clang. >> >> Thus I would like to ask what the LLVM community needs to see from us >> before SPIRV-LLVM can be accepted as an external tool, if at all. >> >> Thanks, >> >> Tomeu >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >
Ronan KERYELL via llvm-dev
2018-Feb-21 11:16 UTC
[llvm-dev] SPIRV-LLVM as an external tool
>>>>> On Wed, 21 Feb 2018 09:58:34 +0000, Nicholas Wilson via llvm-dev <llvm-dev at lists.llvm.org> said:Nicholas> I agree that having the conversion live as part of LLVM is Nicholas> the best option going forward. Yes. :-) Nicholas> libclc is an library implementation of the OpenCL Nicholas> builtins and I think its fine for that aspect of the Nicholas> converter, i.e. consumption of SPIR-V into LLVM IR in Nicholas> preparation for code generation by LLVM by its various Nicholas> backends as a library to link the generated code to, to Nicholas> live in an external repo. Nicholas> This is fine because SPIR<->LLVM IR is trivial, Nicholas> SPIR-V<->LLVM is not. LLVM IR -> SPIR is not that trivial because for example SPIR 2.0 compliant bitcode requires to be encoded into LLVM 3.4 bitcode format, which is not obvious from, say, LLVM 7. By the way there is a market for another open-source project: generic down-grader from modern bitcode to anything older. :-) There are some specific versions flying around, but this is another story... -- Ronan KERYELL
Anastasia Stulova via llvm-dev
2018-Feb-21 15:00 UTC
[llvm-dev] SPIRV-LLVM as an external tool
> I also think that having it in an external project will greatly reduce the visibility in comparison to a target. How many of you have cloned libclc for example?The idea is to have it as a tool in the LLVM project space. This means it will become a part of the overall LLVM project but as an optional code base to the main LLVM backend code base. I don't think this affect visibility that much. For example, Clang is also a tool project of LLVM and it's just as popular as LLVM. It is only a matter of how useful it would be. We are convinced there are enough users of SPIRV converter at the moment for this to be adopted as a component of the LLVM overall infrastructure. Cheers, Anastasia ________________________________ From: llvm-dev <llvm-dev-bounces at lists.llvm.org> on behalf of Nicholas Wilson via llvm-dev <llvm-dev at lists.llvm.org> Sent: 21 February 2018 09:58 To: Tomeu Vizoso Cc: LLVM Developers Subject: Re: [llvm-dev] SPIRV-LLVM as an external tool I agree that having the conversion live as part of LLVM is the best option going forward. libclc is an library implementation of the OpenCL builtins and I think its fine for that aspect of the converter, i.e. consumption of SPIR-V into LLVM IR in preparation for code generation by LLVM by its various backends as a library to link the generated code to, to live in an external repo. This is fine because SPIR<->LLVM IR is trivial, SPIR-V<->LLVM is not. Having the SPIR-V <-> LLVM-IR conversion part of the tool as an external repo as opposed to the llvm would make it very difficult to use as a fontend: First the changes to the build pipeline are a hinderance. In my fork of SPIRV-LLVM the backend is a target because I couldn’t get CMake to link the libraries to LDC as a support library. This would be made even more complicated with at external project. I would probably resort to invoking an external program despite integrated conversion being the best solution. Secondly (unless you've changed things) all of the builtins are not intrinsics, but C++ Itanium with extensions mangled. This makes it so that anyone not implementing a C++ frontend has to deal with it, presumably by hardcoding the names since C++ is very non-trivial. Even for projects like LDC (the LLVM based D compiler) which have a C++ mangler, we still have to hardcode the names because Windows does not use Itanium and the compiler is not set up to do two different forms of C++ mangling, and even if that were possible the non-standard extensions make it infeasible. Whereas any LLVM frontend worth is salt has a way of exposing intrinsics. The backend as it currently stands is not based on tablegen and that is required AFAIU to support intrinsics. I have tablegen descriptions of the SPIR-V format mostly complete (all core instruction relevant to OpenCL + most of the ones for Vulkan, + all the OpenCL extension instructions) though not the intrinsics. However I didn’t get around to using them because I have more important things to work on in the front end. Then there is the question of what to do about Vulkan? I would love to have support for Vulkan in LLVM. That would be relatively simple to set up for with SPIRV as a target, even more so with a tablegen based infrastructure. I have no idea how that would work as an external project. I also think that having it in an external project will greatly reduce the visibility in comparison to a target. How many of you have cloned libclc for example? Not many I suspect. Probably because you are not working on implementing OpenCL for a backend. And while libclc is useful to those implementing OpenCL they are vastly outnumbered by those using OpenCL i.e. producing SPIR/SPIR-V. My usage of SPIR-V is I generate it from my source files though LDC/LLVM then I hand it off to the OpenCL implementation. Nowhere in that pipeline do I need something like libclc. Now libclc may be used in the implementation but that’s a different copy of LLVM to the one that produced it. TL;DR. This really needs to be a target so that we can use intrinsics, have it as part of regular LLVM releases so that it can be used by many frontends. Having an external project for assisting backends consuming SPIR-V (a la libclc) is fine but that shouldn’t mean that the processes used for generating SPIR-V need to go into a separate project. Nic> On 21 Feb 2018, at 4:15 pm, Tomeu Vizoso via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > Hi, > > for a few months already I have been asking around for opinions on how > people could best work together on Khronos' SPIR-V <-> LLVM-IR converter > and some consensus seems to have formed. > > Most of the people I talked to favored having the converter become an > external tool within the LLVM project, similar to libclc. I think that > the LLVM project's processes, infrastructure and community form the best > place for this collaboration to happen. > > I hope that having the converter as part of LLVM can help expand LLVM's > value proposition in heterogeneous computing, in part by complementing > the OpenCL C backend in clang. > > Thus I would like to ask what the LLVM community needs to see from us > before SPIRV-LLVM can be accepted as an external tool, if at all. > > Thanks, > > Tomeu > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev_______________________________________________ LLVM Developers mailing list llvm-dev at lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180221/38da9ee4/attachment.html>
On 02/21/2018 12:15 AM, Tomeu Vizoso via llvm-dev wrote:> Hi, > > for a few months already I have been asking around for opinions on how > people could best work together on Khronos' SPIR-V <-> LLVM-IR converter > and some consensus seems to have formed. > > Most of the people I talked to favored having the converter become an > external tool within the LLVM project, similar to libclc. I think that > the LLVM project's processes, infrastructure and community form the best > place for this collaboration to happen. > > I hope that having the converter as part of LLVM can help expand LLVM's > value proposition in heterogeneous computing, in part by complementing > the OpenCL C backend in clang. > > Thus I would like to ask what the LLVM community needs to see from us > before SPIRV-LLVM can be accepted as an external tool, if at all. >Does the external tool exist already? I recommend starting by posting the code publicly and then ensuring that going forward all development and discussions about the tool are happening in a public space. It's important to demonstrate that there is a developer community backing the tool and that it won't be abandoned if it is added as an llvm sub-project. -Tom> Thanks, > > Tomeu > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Pierre Moreau via llvm-dev
2018-Feb-21 23:34 UTC
[llvm-dev] SPIRV-LLVM as an external tool
On 2018-02-21 — 14:55, Tom Stellard via llvm-dev wrote:> On 02/21/2018 12:15 AM, Tomeu Vizoso via llvm-dev wrote: > > Hi, > > > > for a few months already I have been asking around for opinions on how > > people could best work together on Khronos' SPIR-V <-> LLVM-IR converter > > and some consensus seems to have formed. > > > > Most of the people I talked to favored having the converter become an > > external tool within the LLVM project, similar to libclc. I think that > > the LLVM project's processes, infrastructure and community form the best > > place for this collaboration to happen. > > > > I hope that having the converter as part of LLVM can help expand LLVM's > > value proposition in heterogeneous computing, in part by complementing > > the OpenCL C backend in clang. > > > > Thus I would like to ask what the LLVM community needs to see from us > > before SPIRV-LLVM can be accepted as an external tool, if at all. > > > > Does the external tool exist already? I recommend starting by posting > the code publicly and then ensuring that going forward all development > and discussions about the tool are happening in a public space.The code is publicly available at https://github.com/pierremoreau/llvm-spirv. It is based on https://github.com/KhronosGroup/SPIRV-LLVM/ and https://github.com/thewilsonator/llvm-target-spirv/, stripped off of everything not related to SPIR-V and updated to work with llvm HEAD. (Some of the commits were lost on the way, so we are in the process of redoing the stripping and updating.) One user of that external tool will be Mesa [1] (for its OpenCL implementation named clover) for 1. enabling clCreateProgramWithIL(KHR)? on AMD cards through clover, by translating SPIR-V to LLVM IR; (patches under review [2]) 2. allowing additional drivers (nouveau, freedreno, etnativ) to be supported by clover, by compiling OpenCL C (via LLVM IR) to SPIR-V (and then translating from SPIR-V to NIR). Regards, Pierre [1]: https://www.mesa3d.org/ [2]: https://patchwork.freedesktop.org/series/38728/> It's important to demonstrate that there is a developer community backing > the tool and that it won't be abandoned if it is added as an llvm > sub-project. > > -Tom > > > Thanks, > > > > Tomeu > > _______________________________________________ > > LLVM Developers mailing list > > llvm-dev at lists.llvm.org > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180222/691a2847/attachment.sig>