Timothy Keith via llvm-dev
2020-Jun-12 14:25 UTC
[llvm-dev] [flang-dev] [cfe-dev] [RFC] Refactor Clang: move frontend/driver/diagnostics code to LLVM
For those of us not familiar with clang internals, it would be helpful if you
could describe the parts of clang that you're considering sharing and
explain what existing code they would replace in flang (if any) and what
benefits we gain by sharing them.
In particular, these were mentioned previously: DiagnosticsEngine,
SourceManager, SourceLocation, FileManager, VFS
Thanks,
Tim
On 6/12/20, 3:25 AM, "flang-dev on behalf of Andrzej Warzynski via
flang-dev" <flang-dev-bounces at lists.llvm.org on behalf of flang-dev
at lists.llvm.org> wrote:
External email: Use caution opening links or attachments
FYI: this RFC will be discussed in the next Flang Technical Community
call: http://lists.llvm.org/pipermail/flang-dev/2020-June/000379.html
-Andrzej
On 11/06/2020 23:06, Doerfert, Johannes wrote:
> On 6/11/20 4:04 PM, James Y Knight wrote:
>> I think the expectation is that LLVM remains at the bottom of the
>> dependency tree, with frontend-support depending on LLVM, and Clang
and
>> Flang depending on frontend-support (and LLVM).
>>
>> Not everything which makes sense to share between clang and flang
makes
>> sense to be part of llvm core. E.g., implementation of a
GCC-compatible
>> command-line compiler-driver seems like it would be very much
out-of-place
>> in llvm core.
>
> That clears things up, thanks a lot. So we could have a
frontend-support
>
> subproject for non-LLVM(-IR) related things and `llvm/lib/Frontend` for
>
> LLVM(-IR) related stuff, e.g., IR generation parts.
>
>
> Thanks,
>
> Johannes
>
>
>> On Thu, Jun 11, 2020 at 10:28 AM Doerfert, Johannes via cfe-dev
<
>> cfe-dev at lists.llvm.org> wrote:
>>
>>> On 6/11/20 3:32 AM, Andrzej Warzynski wrote:
>>>>
>>>> On 11/06/2020 00:49, Michael Kruse wrote:
>>>>> Am Mi., 10. Juni 2020 um 10:04 Uhr schrieb Doerfert,
Johannes via
>>>>> flang-dev <flang-dev at lists.llvm.org>:
>>>>>> I'm not against a subproject *but* if we also
move the existing
>>>>>> llvm/lib/Frontend stuff, that would introduce a
dependence from
>>>>>> llvm-core to this project, which I think is
uncommon. We could also
>>>>>> have
>>>>>> both. At the end of the day it depends on the
benefit we would get from
>>>>>> an independent subproject.
>>>>> If we have non-conditional dependencies
llvm-core->frontend-support
>>>>> and frontend-support->llvm-core, what advantage is
there left of
>>>>> making frontend-support a subproject? Both of them are
required all
>>>>> the time.
>>>>>
>>>>> Michael
>>>>>
>>>> Thanks for the feedback! It sounds like we should keep
>>>> llvm/lib/Frontend in LLVM and `frontend-support` would be a
new
>>>> subproject for things unrelated to LLVM IR.
>>> Sure, if we expect usage of such a subproject without
llvm-core. (Maybe
>>> there are other reasons to separate it but I'm not sure I
understood them.)
>>>
>>>
>>> Thanks,
>>>
>>> Johannes
>>>
>>>
>>>> -Andrzej
>>>
>>> _______________________________________________
>>> cfe-dev mailing list
>>> cfe-dev at lists.llvm.org
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>
>
_______________________________________________
flang-dev mailing list
flang-dev at lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev
Andrzej Warzynski via llvm-dev
2020-Jun-12 16:53 UTC
[llvm-dev] [flang-dev] [cfe-dev] [RFC] Refactor Clang: move frontend/driver/diagnostics code to LLVM
Hi Tim, The key idea behind this refactoring is to re-use the compiler driver code [1] that's currently part of Clang. We initially thought that we'd be re-using bits of Clang's frontend-driver too, but from the feedback so far it looks like writing those bits from scratch (e.g. CompilerInstance or CompilerInvocation) would be better. The main benefit of re-using Clang's design is the flexibility that Flang will inherit at a relatively low cost. It will make adding new compiler flags, driver diagnostics or fine-tuning compilation pipelines really easy and powerful. It will enable compilation of mixed-source projects (e.g. C++ + Fortran, perhaps Rust + Fortran one day?). Also, Flang will gain access to Clang’s Toolchains [5], so adoption on new platforms should be more straightforward. And, on top of that, we could start bringing up infrastructure for compiler plugins so that our users can extend Flang as they do Clang. None of this will happen overnight, but given how popular, powerful and widely adopted Clang is, it feels like a good first step. Currently we are focusing on implementing the Flang driver in terms of libclangDriver (which we want to refactor and move out of Clang first). libclangDriver requires some infrastructure code and Diagnostics [2] are a good example. The interface is defined in one header file in Clang [3] (see DiagnosticsEngine, DiagnosticBuilder, DiagnosticConsumer). For now we will only use them for the Flang driver. Wider adoption in Flang would IMHO be great, but that's outside the scope of this RFC and is not required for this to work. The actual diagnostics are defined in TableGen files - mechanism that we also want to adopt. Diagnostics for the driver are defined in [4] (some adoption will be needed there too). In other words - this will only introduce a new Flang driver implemented in terms of `frontend-support` (the new subproject) and hopefully replace the current Flang driver. I can't think of any small/big changes to the Flang codebase that would be required for this. I might be missing something though! As for SourceManager, SourceLocation, FileManager, VFS - to be perfectly honest I think that we need to make more progress with our proof-of-concept implementation before we are ready to discuss these in more detail. We will sent separate RFCs once we are ready. I appreciate that there's not that many technical details here, but we are still working these out. -Andrzej [1] https://clang.llvm.org/docs/DriverInternals.html#driver-design-internals [2] https://clang.llvm.org/diagnostics.html [3] https://github.com/llvm/llvm-project/blob/release/10.x/clang/include/clang/Basic/Diagnostic.h [4] https://github.com/llvm/llvm-project/blob/release/10.x/clang/include/clang/Basic/DiagnosticDriverKinds.td [5] https://github.com/llvm/llvm-project/bloc/release/10.x/clang/lib/Driver/ToolChains On 12/06/2020 15:25, Timothy Keith wrote:> For those of us not familiar with clang internals, it would be helpful if you could describe the parts of clang that you're considering sharing and explain what existing code they would replace in flang (if any) and what benefits we gain by sharing them. > > In particular, these were mentioned previously: DiagnosticsEngine, SourceManager, SourceLocation, FileManager, VFS > > Thanks, > Tim > > On 6/12/20, 3:25 AM, "flang-dev on behalf of Andrzej Warzynski via flang-dev" <flang-dev-bounces at lists.llvm.org on behalf of flang-dev at lists.llvm.org> wrote: > > External email: Use caution opening links or attachments > > > FYI: this RFC will be discussed in the next Flang Technical Community > call: http://lists.llvm.org/pipermail/flang-dev/2020-June/000379.html > > -Andrzej > > On 11/06/2020 23:06, Doerfert, Johannes wrote: > > On 6/11/20 4:04 PM, James Y Knight wrote: > >> I think the expectation is that LLVM remains at the bottom of the > >> dependency tree, with frontend-support depending on LLVM, and Clang and > >> Flang depending on frontend-support (and LLVM). > >> > >> Not everything which makes sense to share between clang and flang makes > >> sense to be part of llvm core. E.g., implementation of a GCC-compatible > >> command-line compiler-driver seems like it would be very much out-of-place > >> in llvm core. > > > > That clears things up, thanks a lot. So we could have a frontend-support > > > > subproject for non-LLVM(-IR) related things and `llvm/lib/Frontend` for > > > > LLVM(-IR) related stuff, e.g., IR generation parts. > > > > > > Thanks, > > > > Johannes > > > > > >> On Thu, Jun 11, 2020 at 10:28 AM Doerfert, Johannes via cfe-dev < > >> cfe-dev at lists.llvm.org> wrote: > >> > >>> On 6/11/20 3:32 AM, Andrzej Warzynski wrote: > >>>> > >>>> On 11/06/2020 00:49, Michael Kruse wrote: > >>>>> Am Mi., 10. Juni 2020 um 10:04 Uhr schrieb Doerfert, Johannes via > >>>>> flang-dev <flang-dev at lists.llvm.org>: > >>>>>> I'm not against a subproject *but* if we also move the existing > >>>>>> llvm/lib/Frontend stuff, that would introduce a dependence from > >>>>>> llvm-core to this project, which I think is uncommon. We could also > >>>>>> have > >>>>>> both. At the end of the day it depends on the benefit we would get from > >>>>>> an independent subproject. > >>>>> If we have non-conditional dependencies llvm-core->frontend-support > >>>>> and frontend-support->llvm-core, what advantage is there left of > >>>>> making frontend-support a subproject? Both of them are required all > >>>>> the time. > >>>>> > >>>>> Michael > >>>>> > >>>> Thanks for the feedback! It sounds like we should keep > >>>> llvm/lib/Frontend in LLVM and `frontend-support` would be a new > >>>> subproject for things unrelated to LLVM IR. > >>> Sure, if we expect usage of such a subproject without llvm-core. (Maybe > >>> there are other reasons to separate it but I'm not sure I understood them.) > >>> > >>> > >>> Thanks, > >>> > >>> Johannes > >>> > >>> > >>>> -Andrzej > >>> > >>> _______________________________________________ > >>> cfe-dev mailing list > >>> cfe-dev at lists.llvm.org > >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev > >>> > > > _______________________________________________ > flang-dev mailing list > flang-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev >
Andrzej Warzynski via llvm-dev
2020-Aug-06 14:45 UTC
[llvm-dev] [flang-dev] [cfe-dev] [RFC] Refactor Clang: move frontend/driver/diagnostics code to LLVM
Just a small FYI, I've restarted this discussion separately on cfe-dev (required refactoring within Clang) and flang-dev (changes in Flang): * cfe-dev: http://lists.llvm.org/pipermail/cfe-dev/2020-July/066393.html * flang-dev: http://lists.llvm.org/pipermail/flang-dev/2020-July/000470.html -Andrzej On 12/06/2020 17:53, Andrzej Warzynski via llvm-dev wrote:> Hi Tim, > > The key idea behind this refactoring is to re-use the compiler driver > code [1] that's currently part of Clang. We initially thought that we'd > be re-using bits of Clang's frontend-driver too, but from the feedback > so far it looks like writing those bits from scratch (e.g. > CompilerInstance or CompilerInvocation) would be better. > > The main benefit of re-using Clang's design is the flexibility that > Flang will inherit at a relatively low cost. It will make adding new > compiler flags, driver diagnostics or fine-tuning compilation pipelines > really easy and powerful. It will enable compilation of mixed-source > projects (e.g. C++ + Fortran, perhaps Rust + Fortran one day?). Also, > Flang will gain access to Clang’s Toolchains [5], so adoption on new > platforms should be more straightforward. And, on top of that, we could > start bringing up infrastructure for compiler plugins so that our users > can extend Flang as they do Clang. None of this will happen overnight, > but given how popular, powerful and widely adopted Clang is, it feels > like a good first step. > > Currently we are focusing on implementing the Flang driver in terms of > libclangDriver (which we want to refactor and move out of Clang first). > libclangDriver requires some infrastructure code and Diagnostics [2] are > a good example. The interface is defined in one header file in Clang [3] > (see DiagnosticsEngine, DiagnosticBuilder, DiagnosticConsumer). For now > we will only use them for the Flang driver. Wider adoption in Flang > would IMHO be great, but that's outside the scope of this RFC and is not > required for this to work. The actual diagnostics are defined in > TableGen files - mechanism that we also want to adopt. Diagnostics for > the driver are defined in [4] (some adoption will be needed there too). > > In other words - this will only introduce a new Flang driver implemented > in terms of `frontend-support` (the new subproject) and hopefully > replace the current Flang driver. I can't think of any small/big changes > to the Flang codebase that would be required for this. I might be > missing something though! As for SourceManager, SourceLocation, > FileManager, VFS - to be perfectly honest I think that we need to make > more progress with our proof-of-concept implementation before we are > ready to discuss these in more detail. We will sent separate RFCs once > we are ready. > > I appreciate that there's not that many technical details here, but we > are still working these out. > > -Andrzej > > > [1] > https://clang.llvm.org/docs/DriverInternals.html#driver-design-internals > [2] https://clang.llvm.org/diagnostics.html > [3] > https://github.com/llvm/llvm-project/blob/release/10.x/clang/include/clang/Basic/Diagnostic.h > > [4] > https://github.com/llvm/llvm-project/blob/release/10.x/clang/include/clang/Basic/DiagnosticDriverKinds.td > > [5] > https://github.com/llvm/llvm-project/bloc/release/10.x/clang/lib/Driver/ToolChains > > > On 12/06/2020 15:25, Timothy Keith wrote: >> For those of us not familiar with clang internals, it would be helpful >> if you could describe the parts of clang that you're considering >> sharing and explain what existing code they would replace in flang (if >> any) and what benefits we gain by sharing them. >> >> In particular, these were mentioned previously: DiagnosticsEngine, >> SourceManager, SourceLocation, FileManager, VFS >> >> Thanks, >> Tim >> >> On 6/12/20, 3:25 AM, "flang-dev on behalf of Andrzej Warzynski via >> flang-dev" <flang-dev-bounces at lists.llvm.org on behalf of >> flang-dev at lists.llvm.org> wrote: >> >> External email: Use caution opening links or attachments >> >> >> FYI: this RFC will be discussed in the next Flang Technical >> Community >> call: >> http://lists.llvm.org/pipermail/flang-dev/2020-June/000379.html >> >> -Andrzej >> >> On 11/06/2020 23:06, Doerfert, Johannes wrote: >> > On 6/11/20 4:04 PM, James Y Knight wrote: >> >> I think the expectation is that LLVM remains at the bottom of the >> >> dependency tree, with frontend-support depending on LLVM, and >> Clang and >> >> Flang depending on frontend-support (and LLVM). >> >> >> >> Not everything which makes sense to share between clang and >> flang makes >> >> sense to be part of llvm core. E.g., implementation of a >> GCC-compatible >> >> command-line compiler-driver seems like it would be very much >> out-of-place >> >> in llvm core. >> > >> > That clears things up, thanks a lot. So we could have a >> frontend-support >> > >> > subproject for non-LLVM(-IR) related things and >> `llvm/lib/Frontend` for >> > >> > LLVM(-IR) related stuff, e.g., IR generation parts. >> > >> > >> > Thanks, >> > >> > Johannes >> > >> > >> >> On Thu, Jun 11, 2020 at 10:28 AM Doerfert, Johannes via cfe-dev < >> >> cfe-dev at lists.llvm.org> wrote: >> >> >> >>> On 6/11/20 3:32 AM, Andrzej Warzynski wrote: >> >>>> >> >>>> On 11/06/2020 00:49, Michael Kruse wrote: >> >>>>> Am Mi., 10. Juni 2020 um 10:04 Uhr schrieb Doerfert, >> Johannes via >> >>>>> flang-dev <flang-dev at lists.llvm.org>: >> >>>>>> I'm not against a subproject *but* if we also move the >> existing >> >>>>>> llvm/lib/Frontend stuff, that would introduce a dependence >> from >> >>>>>> llvm-core to this project, which I think is uncommon. We >> could also >> >>>>>> have >> >>>>>> both. At the end of the day it depends on the benefit we >> would get from >> >>>>>> an independent subproject. >> >>>>> If we have non-conditional dependencies >> llvm-core->frontend-support >> >>>>> and frontend-support->llvm-core, what advantage is there >> left of >> >>>>> making frontend-support a subproject? Both of them are >> required all >> >>>>> the time. >> >>>>> >> >>>>> Michael >> >>>>> >> >>>> Thanks for the feedback! It sounds like we should keep >> >>>> llvm/lib/Frontend in LLVM and `frontend-support` would be a new >> >>>> subproject for things unrelated to LLVM IR. >> >>> Sure, if we expect usage of such a subproject without >> llvm-core. (Maybe >> >>> there are other reasons to separate it but I'm not sure I >> understood them.) >> >>> >> >>> >> >>> Thanks, >> >>> >> >>> Johannes >> >>> >> >>> >> >>>> -Andrzej >> >>> >> >>> _______________________________________________ >> >>> cfe-dev mailing list >> >>> cfe-dev at lists.llvm.org >> >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev >> >>> >> > >> _______________________________________________ >> flang-dev mailing list >> flang-dev at lists.llvm.org >> https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev >> > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev