llvm dev - Aug 2017 - Extending TableGen's 'foreach' to work with 'multiclass' and 'defm'

On 08/23/2017 12:06 PM, Krzysztof Parzyszek via llvm-dev
wrote:
> On 8/23/2017 11:58 AM, Hal Finkel via llvm-dev wrote:
>> If we want to go down that route, I can certainly imagine a feasible 
>> incremental-transitioning strategy. We could allow TableGen to use an 
>> embedded Python interpreter to generate records based on Python data 
>> structures, and then, combine records from the existing .td files 
>> with those generated by the Python code. We'd use the existing 
>> TableGen plugins (which we may need to continue to use regardless, 
>> compared to writing Python, for performance reasons), and so we could 
>> incrementally transition existing definitions from .td files to 
>> Python as appropriate.
>
> Would we then eliminate TableGen completely in the long term?
That could also be two separate questions: Would we replace the .td 
input language with Python completely in the long term? Would we rewrite 
the the backends (i.e., TableGen plugins) in Python? I don't yet have an 
opinion on either. I can see advantages to providing Python as input 
language. What do you think?

  -Hal
>
> -Krzysztof
>
-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory

On 8/23/2017 12:21 PM, Hal Finkel wrote:>
> That could also be two separate questions: Would we replace the .td 
> input language with Python completely in the long term? Would we rewrite 
> the the backends (i.e., TableGen plugins) in Python? I don't yet have
an
> opinion on either. I can see advantages to providing Python as input 
> language. What do you think?
If we want to undertake incorporating Python into the TableGen pipeline, 
then completely replacing TableGen with Python sounds like a logical 
long-term goal.  Everything that TableGen can do can be done in Python 
as well, plus Python offers nearly unlimited flexibility.

-Krzysztof

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, 
hosted by The Linux Foundation

On 23 Aug 2017, at 18:21, Hal Finkel via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> 
> On 08/23/2017 12:06 PM, Krzysztof Parzyszek via llvm-dev wrote:
>> On 8/23/2017 11:58 AM, Hal Finkel via llvm-dev wrote:
>>> If we want to go down that route, I can certainly imagine a
feasible incremental-transitioning strategy. We could allow TableGen to use an
embedded Python interpreter to generate records based on Python data structures,
and then, combine records from the existing .td files with those generated by
the Python code. We'd use the existing TableGen plugins (which we may need
to continue to use regardless, compared to writing Python, for performance
reasons), and so we could incrementally transition existing definitions from .td
files to Python as appropriate.
>> 
>> Would we then eliminate TableGen completely in the long term?
> 
> That could also be two separate questions: Would we replace the .td input
language with Python completely in the long term? Would we rewrite the the
backends (i.e., TableGen plugins) in Python? I don't yet have an opinion on
either. I can see advantages to providing Python as input language. What do you
think?
Replacing TableGen with general purpose language X runs into the issue of
bikeshedding what X should be.  I’d be very much opposed to Python because:

 - It’s a large external dependency for the build (there’s no chance of FreeBSD
shipping Python in the base system, for example, so we’d have to import the
Python-generated files on each import, which would be annoying)

 - The language has had one backwards-incompatible break that it’s taken over a
decade to recover from, I have little confidence that it will remain compatible
going forward

 - It seems to encourage terrible code (I have yet to be presented with a piece
of allegedly working Python software that I have not had to fix at least one bug
in - git-imerge was almost an exception, but sadly not quite).

 - It intentionally doesn’t support tail recursion optimisation and imposes
arbitrary stack depth limits, which forces some convoluted coding styles.

More generally, I’m not sure about the underlying goal.  We already have one
solid general-purpose language in LLVM: C++.  A lot of the things that we
currently use more complex TableGen programming practices for now would be good
uses for the proposed metaclass / reflection APIs in C++21, which seems a more
palatable end goal than a scripting language.

There are also external tools that both produce and consume the TableGen
sources.  Having a language that is *not* a general-purpose programming language
is a feature for these tools, not an obstacle.

David

On 8/23/2017 12:44 PM, David Chisnall wrote:
> I’d be very much opposed to Python because:
> 
>   - It’s a large external dependency for the build (there’s no chance of
FreeBSD shipping Python in the base system, for example, so we’d have to import
the Python-generated files on each import, which would be annoying)
This isn't really specific to replacing TableGen with Python, as much as 
it is a concern against using Python altogether.  The original idea here 
was to add it as an extra tool to aid with the processing of .td files.

I'm guessing that the annoyance would come from the fact that bypassing 
TableGen, and using the pre-existing .inc files is not well supported by 
the build process.  What if this was made easier?

-Krzysztof

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, 
hosted by The Linux Foundation

On 23 August 2017 at 18:21, Hal Finkel via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> That could also be two separate questions: Would we replace the .td input
> language with Python completely in the long term? Would we rewrite the the
> backends (i.e., TableGen plugins) in Python? I don't yet have an
opinion on
> either. I can see advantages to providing Python as input language. What do
> you think?
I see three options it's worth differentiating between due to the
different trade-offs in terms of benefit, disruption, build-time
dependencies etc etc:
1) Keep tablegen as-is, but encourage/allow backends to use a more
powerful language to generate very straight-forward .td. This might
take a template-based approach (e.g. Jinja2), or perhaps just
pretty-printing the preferred data structure in to .td.
2) Embed Python in tablegen (i.e. python will be linked in to the
tablegen binary).
3) Drop tablegen altogether, develop an alternative tool for
generating the necessary .inc files

A number of LLVM users already do something like option 1). e.g. the
Hexagon backend or a variety of companies who offer tools for
generating custom processor cores + compiler support based on
specifying new instructions in a custom DSL.

Best,

Alex

On 08/23/2017 12:44 PM, David Chisnall wrote:
> On 23 Aug 2017, at 18:21, Hal Finkel via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
>>
>> On 08/23/2017 12:06 PM, Krzysztof Parzyszek via llvm-dev wrote:
>>> On 8/23/2017 11:58 AM, Hal Finkel via llvm-dev wrote:
>>>> If we want to go down that route, I can certainly imagine a
feasible incremental-transitioning strategy. We could allow TableGen to use an
embedded Python interpreter to generate records based on Python data structures,
and then, combine records from the existing .td files with those generated by
the Python code. We'd use the existing TableGen plugins (which we may need
to continue to use regardless, compared to writing Python, for performance
reasons), and so we could incrementally transition existing definitions from .td
files to Python as appropriate.
>>> Would we then eliminate TableGen completely in the long term?
>> That could also be two separate questions: Would we replace the .td
input language with Python completely in the long term? Would we rewrite the the
backends (i.e., TableGen plugins) in Python? I don't yet have an opinion on
either. I can see advantages to providing Python as input language. What do you
think?
> Replacing TableGen with general purpose language X runs into the issue of
bikeshedding what X should be.  I’d be very much opposed to Python because:
>
>   - It’s a large external dependency for the build (there’s no chance of
FreeBSD shipping Python in the base system, for example, so we’d have to import
the Python-generated files on each import, which would be annoying)
Given that our regression-testing infrastructure is built on Python, I 
already consider it to be a build dependency (and, as I recall, there 
are parts of the compiler-rt/libc++ builds that depend on it as well). 
This, and a wide user base, motivate my suggestion.
>
>   - The language has had one backwards-incompatible break that it’s taken
over a decade to recover from, I have little confidence that it will remain
compatible going forward
>
>   - It seems to encourage terrible code (I have yet to be presented with a
piece of allegedly working Python software that I have not had to fix at least
one bug in - git-imerge was almost an exception, but sadly not quite).
>
>   - It intentionally doesn’t support tail recursion optimisation and
imposes arbitrary stack depth limits, which forces some convoluted coding
styles.
>
> More generally, I’m not sure about the underlying goal.  We already have
one solid general-purpose language in LLVM: C++.  A lot of the things that we
currently use more complex TableGen programming practices for now would be good
uses for the proposed metaclass / reflection APIs in C++21, which seems a more
palatable end goal than a scripting language.
Maybe, but it might be a long time before we can use C++20.
>
> There are also external tools that both produce and consume the TableGen
sources.  Having a language that is *not* a general-purpose programming language
is a feature for these tools, not an obstacle.
I think that this goes both ways. For in-tree targets at least, we want 
the source that is intended to be maintained by a human in tree. So, the 
fact that an external tool can produce TableGen doesn't really solve the 
problem (unless we can also have that tool in tree, and moreover, I 
don't want to encourage each backend to develop their own such tools 
independently).

  -Hal
>
> David
>
-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory