William Moses via llvm-dev
2021-Jan-12 20:57 UTC
[llvm-dev] RFC: Enzyme, Automatic Differentiation for LLVM as an LLVM Incubator Project
Hi all, Automatic differentiation (AD) is a key component in algorithms used in machine learning, scientific computing, and elsewhere. For the last year-and-a-half, the `Enzyme` group have been looking at the practical possibility of doing automatic differentiation as part of the LLVM optimization pipeline. Performing automatic differentiation in LLVM is quite beneficial as it allows all of the languages that lower to LLVM to incorporate automatic differentiation without much additional work. It also allows for automatic differentiation across languages, which is similarly beneficial. One unexpected benefit we found of doing AD at the LLVM-level is that there is a significant performance benefit (4.2x in our tests) to be gained by performing AD after LLVM’s optimization passes [1]. After several months of testing with various users including the Rust [4, 5], C/C++, Julia [6], Fortran, and machine learning communities, we’d like to share LLVM-based automatic differentiation more widely and ask to be considered as an LLVM incubator project. Our code is available here ( https://github.com/wsmoses/Enzyme/tree/master/enzyme) as a plugin for LLVM versions 7 through master. We’ve had weekly meetings for the past several months with folks from MIT, Argonne, Princeton, Google, NVIDIA, and Facebook and welcome anyone who wants to join. Documentation and install instructions for Enzyme is available here: https://enzyme.mit.edu. We have our charter available here: <https://docs.google.com/document/d/10IK2EgZa-4WF0lOSlkND1_cX3IQLAxEVSOWqbQzNpcs/edit#> https://docs.google.com/document/d/10IK2EgZa-4WF0lOSlkND1_cX3IQLAxEVSOWqbQzNpcs/edit# Performing automatic differentiation inside of LLVM presents several interesting technical questions, which we’ve explored with the community in a poster and SRC talk at the 2020 US LLVM Dev Meeting [2, 3]. The Enzyme team [1] https://proceedings.neurips.cc/paper/2020/file/9332c513ef44b682e9347822c2e457ac-Paper.pdf [2] https://c.wsmoses.com/posters/Enzyme-llvmdev.pdf [3] https://www.youtube.com/watch?v=auQNFDlaXdM, https://c.wsmoses.com/presentations/enzyme-llvmdev-reduced.pdf [4] https://github.com/tiberiusferreira/oxide-enzyme https://github.com/bytesnake/oxide-enzyme, [5] https://internals.rust-lang.org/t/automatic-differentiation-differential-programming-via-llvm/13188 [6] https://github.com/wsmoses/Enzyme.jl -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210112/84509725/attachment.html>
Mehdi AMINI via llvm-dev
2021-Jan-12 21:32 UTC
[llvm-dev] RFC: Enzyme, Automatic Differentiation for LLVM as an LLVM Incubator Project
Hi, Since the project is alright going on for some time, what is the current level of maturity? You're proposing to get through as an incubator project, how far is it from getting at the point where it would integrate the monorepo? How do you see the roadmap on this? Thanks, -- Mehdi On Tue, Jan 12, 2021 at 12:58 PM William Moses via llvm-dev < llvm-dev at lists.llvm.org> wrote:> Hi all, > > Automatic differentiation (AD) is a key component in algorithms used in > machine learning, scientific computing, and elsewhere. > > For the last year-and-a-half, the `Enzyme` group have been looking at the > practical possibility of doing automatic differentiation as part of the > LLVM optimization pipeline. Performing automatic differentiation in LLVM is > quite beneficial as it allows all of the languages that lower to LLVM to > incorporate automatic differentiation without much additional work. It also > allows for automatic differentiation across languages, which is similarly > beneficial. > > One unexpected benefit we found of doing AD at the LLVM-level is that > there is a significant performance benefit (4.2x in our tests) to be gained > by performing AD after LLVM’s optimization passes [1]. > > > After several months of testing with various users including the Rust [4, > 5], C/C++, Julia [6], Fortran, and machine learning communities, we’d like > to share LLVM-based automatic differentiation more widely and ask to be > considered as an LLVM incubator project. > > Our code is available here ( > https://github.com/wsmoses/Enzyme/tree/master/enzyme) as a plugin for > LLVM versions 7 through master. We’ve had weekly meetings for the past > several months with folks from MIT, Argonne, Princeton, Google, NVIDIA, and > Facebook and welcome anyone who wants to join. Documentation and install > instructions for Enzyme is available here: https://enzyme.mit.edu. We > have our charter available here: > > <https://docs.google.com/document/d/10IK2EgZa-4WF0lOSlkND1_cX3IQLAxEVSOWqbQzNpcs/edit#> > https://docs.google.com/document/d/10IK2EgZa-4WF0lOSlkND1_cX3IQLAxEVSOWqbQzNpcs/edit# > > > Performing automatic differentiation inside of LLVM presents several > interesting technical questions, which we’ve explored with the community in > a poster and SRC talk at the 2020 US LLVM Dev Meeting [2, 3]. > > The Enzyme team > > [1] > https://proceedings.neurips.cc/paper/2020/file/9332c513ef44b682e9347822c2e457ac-Paper.pdf > > [2] https://c.wsmoses.com/posters/Enzyme-llvmdev.pdf > > [3] https://www.youtube.com/watch?v=auQNFDlaXdM, > https://c.wsmoses.com/presentations/enzyme-llvmdev-reduced.pdf > > [4] https://github.com/tiberiusferreira/oxide-enzyme > https://github.com/bytesnake/oxide-enzyme, > > [5] > https://internals.rust-lang.org/t/automatic-differentiation-differential-programming-via-llvm/13188 > > [6] https://github.com/wsmoses/Enzyme.jl > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210112/0ece3bba/attachment.html>
Vassil Vassilev via llvm-dev
2021-Jan-14 22:07 UTC
[llvm-dev] RFC: Enzyme, Automatic Differentiation for LLVM as an LLVM Incubator Project
Hi, This would be very useful for the field of high energy physics and data science in general. Having enzyme as part of llvm would reduce the amount of custom code we have in our autodiff tool clad. I can see it being a common ground of a bunch of other tools across languages generating llvm. Thanks for working on making this upstream! Best, Vassil On 1/12/21 10:57 PM, William Moses wrote:> Hi all, > Automatic differentiation (AD) is a key component in algorithms used > in machine learning, scientific computing, and elsewhere. > For the last year-and-a-half, the `Enzyme` group have been looking at > the practical possibility of doing automatic differentiation as part > of the LLVM optimization pipeline. Performing automatic > differentiation in LLVM is quite beneficial as it allows all of the > languages that lower to LLVM to incorporate automatic differentiation > without much additional work. It also allows for automatic > differentiation across languages, which is similarly beneficial. > One unexpected benefit we found of doing AD at the LLVM-level is that > there is a significant performance benefit (4.2x in our tests) to be > gained by performing AD after LLVM’s optimization passes [1]. > > After several months of testing with various users including the Rust > [4, 5], C/C++, Julia [6], Fortran, and machine learning communities, > we’d like to share LLVM-based automatic differentiation more widely > and ask to be considered as an LLVM incubator project. > > > Our code is available here > (https://github.com/wsmoses/Enzyme/tree/master/enzyme > <https://github.com/wsmoses/Enzyme/tree/master/enzyme>) as a plugin > for LLVM versions 7 through master. We’ve had weekly meetings for the > past several months with folks from MIT, Argonne, Princeton, Google, > NVIDIA, and Facebook and welcome anyone who wants to join. > Documentation and install instructions for Enzyme is available here: > https://enzyme.mit.edu <https://enzyme.mit.edu>. We have our charter > available here: > https://docs.google.com/document/d/10IK2EgZa-4WF0lOSlkND1_cX3IQLAxEVSOWqbQzNpcs/edit# > <https://docs.google.com/document/d/10IK2EgZa-4WF0lOSlkND1_cX3IQLAxEVSOWqbQzNpcs/edit#> > > Performing automatic differentiation inside of LLVM presents several > interesting technical questions, which we’ve explored with the > community in a poster and SRC talk at the 2020 US LLVM Dev Meeting [2, 3]. > > > The Enzyme team > > > [1] > https://proceedings.neurips.cc/paper/2020/file/9332c513ef44b682e9347822c2e457ac-Paper.pdf > <https://proceedings.neurips.cc/paper/2020/file/9332c513ef44b682e9347822c2e457ac-Paper.pdf> > > [2] https://c.wsmoses.com/posters/Enzyme-llvmdev.pdf > <https://c.wsmoses.com/posters/Enzyme-llvmdev.pdf> > > [3] https://www.youtube.com/watch?v=auQNFDlaXdM > <https://www.youtube.com/watch?v=auQNFDlaXdM>, > https://c.wsmoses.com/presentations/enzyme-llvmdev-reduced.pdf > <https://c.wsmoses.com/presentations/enzyme-llvmdev-reduced.pdf> > > [4] https://github.com/tiberiusferreira/oxide-enzyme > <https://github.com/tiberiusferreira/oxide-enzyme>https://github.com/bytesnake/oxide-enzyme > <https://github.com/bytesnake/oxide-enzyme>, > > [5] > https://internals.rust-lang.org/t/automatic-differentiation-differential-programming-via-llvm/13188 > <https://internals.rust-lang.org/t/automatic-differentiation-differential-programming-via-llvm/13188> > > [6] https://github.com/wsmoses/Enzyme.jl > <https://github.com/wsmoses/Enzyme.jl> > -- > You received this message because you are subscribed to the Google > Groups "Enzyme AD" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to enzyme-dev+unsubscribe at googlegroups.com > <mailto:enzyme-dev+unsubscribe at googlegroups.com>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/enzyme-dev/CAGPZhf0X%3DRbym02pKmDOhF5wMuYL_UuBcrhaOEUC4e%3D8sESe0g%40mail.gmail.com > <https://groups.google.com/d/msgid/enzyme-dev/CAGPZhf0X%3DRbym02pKmDOhF5wMuYL_UuBcrhaOEUC4e%3D8sESe0g%40mail.gmail.com?utm_medium=email&utm_source=footer>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>.-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210115/4960bbd8/attachment.html>