llvm dev - Aug 2017 - Extending TableGen's 'foreach' to work with 'multiclass' and 'defm'

On 08/22/2017 03:59 AM, Alex Bradbury via llvm-dev
wrote:
> On 21 August 2017 at 13:23, Martin J. O'Riordan via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>> But there is a downside.
>>
>> For each of the above I also have variations that are a result of
different
>> processor and ISA versions, and because of this I have to use
>> ‘multiclass/defm’ to define the descriptions along with ‘Require’
>> predicates.  The same approach does not work with ‘multiclass/defm’
though,
>> because TableGen does not support ‘foreach’ with ‘multiclass/defm’.
>>
>> I have experimented with adapting TableGen to do this, but I am just
not
>> knowledgeable enough about how TableGen works and my attempts have not
been
>> successful.  Perhaps some of the people debating the separation of
>> instruction and patterns topic might have some insight into how
TableGen
>> might be adapted to support ‘foreach’ with ‘multiclass/defm’
definitions and
>> could advise me how I should do this; or maybe the maintainers of
TableGen
>> might consider this something that they would be willing to add to
TableGen
>> in the future?
> Hi Martin, I think this is an interesting topic. I've also run up
> against the limitations of foreach, though for my particular case the
> variable-sized register class work provides a better solution.
>
> I will note that at least one backend (Hexagon) has moved towards
> using TableGen as a fairly 'dumb' data definition language, relying
on
> a separate tool for generating instruction definitions. I'd be curious
> to know if others are using this approach. It'd also imaging that
> using m4/jinja or similar as a .td preprocessor would be a potential
> option for an out-of-tree backend, in cases where TableGen macro
> support and programmability is too weak.
>
> I suppose one question is: would allowing foreach to be used with
> multiclass/defm be sufficient to allow TableGen to be a productive and
> maintainable way of defining complex architectures, or would there be
> a number of other deficiencies that might push you towards larger
> TableGen extensions or using a separate tool or preprocessor?
The fact that you can't use foreach with multiclasses is a bug, and we 
should fix it, if possible, regardless of whether it is the last 
remaining roadblock to handling complex architectures. For situations 
well beyond TableGen's current language capabilities, we have a decision 
to make. We can continue extending TableGen until it can meet those 
needs. Alternatively, we can enable the use of some more-powerful input 
language. For example, we could allow TableGen to embed Python, and then 
use Python in order to generate record definitions.

  -Hal
>
> Best,
>
> Alex
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory

> On Aug 22, 2017, at 14:15, Hal Finkel via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> For situations well beyond TableGen's current language capabilities, we
have a decision to make. We can continue extending TableGen until it can meet
those needs. Alternatively, we can enable the use of some more-powerful input
language. For example, we could allow TableGen to embed Python, and then use
Python in order to generate record definitions.
For a project that’s not LLVM, I recently had the opportunity to replace both
TableGen and *.td files with Python scripts. I found that TableGen’s features
were easily matched by Python’s for loops and the ability to define functions. I
am pretty happy with the approach so far. AMA

This is a lot easier to do in a green field project than in an old project like
LLVM, of course.

Example “.td” file:
https://github.com/stoklund/cretonne/blob/master/lib/cretonne/meta/isa/riscv/encodings.py
<https://github.com/stoklund/cretonne/blob/master/lib/cretonne/meta/isa/riscv/encodings.py>

Thanks,
/jakob
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170823/f1158d81/attachment.html>

On 08/23/2017 11:35 AM, Jakob Stoklund Olesen wrote:
>
>> On Aug 22, 2017, at 14:15, Hal Finkel via llvm-dev 
>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>>
>> For situations well beyond TableGen's current language
capabilities,
>> we have a decision to make. We can continue extending TableGen until 
>> it can meet those needs. Alternatively, we can enable the use of some 
>> more-powerful input language. For example, we could allow TableGen to 
>> embed Python, and then use Python in order to generate record 
>> definitions.
>
> For a project that’s not LLVM, I recently had the opportunity to 
> replace both TableGen and *.td files with Python scripts. I found that 
> TableGen’s features were easily matched by Python’s for loops and the 
> ability to define functions. I am pretty happy with the approach so 
> far. AMA
Interesting, thanks.
>
> This is a lot easier to do in a green field project than in an old 
> project like LLVM, of course.
If we want to go down that route, I can certainly imagine a feasible 
incremental-transitioning strategy. We could allow TableGen to use an 
embedded Python interpreter to generate records based on Python data 
structures, and then, combine records from the existing .td files with 
those generated by the Python code. We'd use the existing TableGen 
plugins (which we may need to continue to use regardless, compared to 
writing Python, for performance reasons), and so we could incrementally 
transition existing definitions from .td files to Python as appropriate.

  -Hal
>
> Example “.td” file: 
>
https://github.com/stoklund/cretonne/blob/master/lib/cretonne/meta/isa/riscv/encodings.py
>
> Thanks,
> /jakob
-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170823/925a7c33/attachment.html>

For what it's worth: I also had very good experiences with python
"specifications" to generate code [1].

Of course it's hard to justify switching all the infrastructure just because
of one missing tablegen feature...

- Matthias

[1] Example spec + template:
https://github.com/libfirm/libfirm/blob/master/scripts/ir_spec.py
<https://github.com/libfirm/libfirm/blob/master/scripts/ir_spec.py>
  https://github.com/libfirm/libfirm/blob/master/scripts/templates/gen_irio.c
> On Aug 23, 2017, at 9:35 AM, Jakob Stoklund Olesen via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> 
> 
>> On Aug 22, 2017, at 14:15, Hal Finkel via llvm-dev <llvm-dev at
lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>> 
>> For situations well beyond TableGen's current language
capabilities, we have a decision to make. We can continue extending TableGen
until it can meet those needs. Alternatively, we can enable the use of some
more-powerful input language. For example, we could allow TableGen to embed
Python, and then use Python in order to generate record definitions.
> 
> For a project that’s not LLVM, I recently had the opportunity to replace
both TableGen and *.td files with Python scripts. I found that TableGen’s
features were easily matched by Python’s for loops and the ability to define
functions. I am pretty happy with the approach so far. AMA
> 
> This is a lot easier to do in a green field project than in an old project
like LLVM, of course.
> 
> Example “.td” file:
https://github.com/stoklund/cretonne/blob/master/lib/cretonne/meta/isa/riscv/encodings.py
<https://github.com/stoklund/cretonne/blob/master/lib/cretonne/meta/isa/riscv/encodings.py>
> 
> Thanks,
> /jakob
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170823/ee1a7e64/attachment.html>