llvm dev - Aug 2017 - Extending TableGen's 'foreach' to work with 'multiclass' and 'defm'

On 08/23/2017 12:44 PM, David Chisnall wrote:
> On 23 Aug 2017, at 18:21, Hal Finkel via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
>>
>> On 08/23/2017 12:06 PM, Krzysztof Parzyszek via llvm-dev wrote:
>>> On 8/23/2017 11:58 AM, Hal Finkel via llvm-dev wrote:
>>>> If we want to go down that route, I can certainly imagine a
feasible incremental-transitioning strategy. We could allow TableGen to use an
embedded Python interpreter to generate records based on Python data structures,
and then, combine records from the existing .td files with those generated by
the Python code. We'd use the existing TableGen plugins (which we may need
to continue to use regardless, compared to writing Python, for performance
reasons), and so we could incrementally transition existing definitions from .td
files to Python as appropriate.
>>> Would we then eliminate TableGen completely in the long term?
>> That could also be two separate questions: Would we replace the .td
input language with Python completely in the long term? Would we rewrite the the
backends (i.e., TableGen plugins) in Python? I don't yet have an opinion on
either. I can see advantages to providing Python as input language. What do you
think?
> Replacing TableGen with general purpose language X runs into the issue of
bikeshedding what X should be.  I’d be very much opposed to Python because:
>
>   - It’s a large external dependency for the build (there’s no chance of
FreeBSD shipping Python in the base system, for example, so we’d have to import
the Python-generated files on each import, which would be annoying)
Given that our regression-testing infrastructure is built on Python, I 
already consider it to be a build dependency (and, as I recall, there 
are parts of the compiler-rt/libc++ builds that depend on it as well). 
This, and a wide user base, motivate my suggestion.
>
>   - The language has had one backwards-incompatible break that it’s taken
over a decade to recover from, I have little confidence that it will remain
compatible going forward
>
>   - It seems to encourage terrible code (I have yet to be presented with a
piece of allegedly working Python software that I have not had to fix at least
one bug in - git-imerge was almost an exception, but sadly not quite).
>
>   - It intentionally doesn’t support tail recursion optimisation and
imposes arbitrary stack depth limits, which forces some convoluted coding
styles.
>
> More generally, I’m not sure about the underlying goal.  We already have
one solid general-purpose language in LLVM: C++.  A lot of the things that we
currently use more complex TableGen programming practices for now would be good
uses for the proposed metaclass / reflection APIs in C++21, which seems a more
palatable end goal than a scripting language.
Maybe, but it might be a long time before we can use C++20.
>
> There are also external tools that both produce and consume the TableGen
sources.  Having a language that is *not* a general-purpose programming language
is a feature for these tools, not an obstacle.
I think that this goes both ways. For in-tree targets at least, we want 
the source that is intended to be maintained by a human in tree. So, the 
fact that an external tool can produce TableGen doesn't really solve the 
problem (unless we can also have that tool in tree, and moreover, I 
don't want to encourage each backend to develop their own such tools 
independently).

  -Hal
>
> David
>
-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory

The compilers I maintain are for our own custom hardware processor designs, so
it is normal for the hardware and tools to be developed in conjunction with each
other; and a question I am asked about twice a year is:

  "can we not just generate the tool-chain from the machine
description?"

Seems simple enough, no?  The object being to have a single definitive statement
of the machine, and from that derive the RTL, silicon, documentation, assembler,
compiler, debugger, simulator, etc.  It’s a neat objective, but not well
realised.

There have been attempts in the past, but they always seem to fizzle out after a
while, often because they begin as University PhD research topics, and after the
original dissertation is completed, they just seem to die.

'ArchC' was an example, and 'AACGen', but I haven't seen any
progress on this since 2013, and even then it was using LLVM v2.7 (or possible
older).  I just had a quick look at the website for ArchC, and it seems that it
is still stuck in 2013 :-(

Hal's comment "unless we can also have that tool in tree" I think
is important, because if an alternative to TableGen is devised, then having its
source within the LLVM tree ensures that it will be maintained and will develop
over time.

	MartinO

-----Original Message-----
From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of Hal
Finkel via llvm-dev
Sent: 23 August 2017 19:45
>> There are also external tools that both produce and consume the
TableGen sources.  Having
>> a language that is *not* a general-purpose programming language is a
feature for these
>> tools, not an obstacle.
>
> I think that this goes both ways. For in-tree targets at least, we want 
> the source that is intended to be maintained by a human in tree. So, the 
> fact that an external tool can produce TableGen doesn't really solve
the
> problem (unless we can also have that tool in tree, and moreover, I 
> don't want to encourage each backend to develop their own such tools 
> independently).
>
>   -Hal

On 24 Aug 2017, at 11:20, Martin J. O'Riordan <MartinO at theheart.ie>
wrote:
> 
> Seems simple enough, no?  The object being to have a single definitive
statement of the machine, and from that derive the RTL, silicon, documentation,
assembler, compiler, debugger, simulator, etc.  It’s a neat objective, but not
well realised.
> 
> There have been attempts in the past, but they always seem to fizzle out
after a while, often because they begin as University PhD research topics, and
after the original dissertation is completed, they just seem to die.
The Synopsis toolchain with their Lisa HDL can generate a TableGen back end for
LLVM from the instruction descriptions.

David

Hi Martin et al,
>
>   "can we not just generate the tool-chain from the machine
description?"
>
At Synopsys, we do have and provide such tools (and they also use LLVM).

For more information, you can check:
<https://www.synopsys.com/dw/ipdir.php?ds=asip-designer>
or you can contact me directly.

Greetings,

Jeroen Dobbelaere

> -----Original Message-----
> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of
> Martin J. O'Riordan via llvm-dev
> Sent: Thursday, August 24, 2017 12:21
> To: 'Hal Finkel' <hfinkel at anl.gov>; 'David
Chisnall'
> <David.Chisnall at cl.cam.ac.uk>; 'LLVM Developers'
<llvm-dev at lists.llvm.org>
> Subject: Re: [llvm-dev] Extending TableGen's 'foreach' to work
with
> 'multiclass' and 'defm'
> 
> The compilers I maintain are for our own custom hardware processor
> designs, so it is normal for the hardware and tools to be developed in
> conjunction with each other; and a question I am asked about twice a year
is:
> 
>   "can we not just generate the tool-chain from the machine
description?"
> 
> Seems simple enough, no?  The object being to have a single definitive
> statement of the machine, and from that derive the RTL, silicon,
> documentation, assembler, compiler, debugger, simulator, etc.  It’s a neat
> objective, but not well realised.
> 
> There have been attempts in the past, but they always seem to fizzle out
> after a while, often because they begin as University PhD research topics,
> and after the original dissertation is completed, they just seem to die.
> 
> 'ArchC' was an example, and 'AACGen', but I haven't
seen any progress on
> this since 2013, and even then it was using LLVM v2.7 (or possible older). 
I
> just had a quick look at the website for ArchC, and it seems that it is
still stuck
> in 2013 :-(
> 
> Hal's comment "unless we can also have that tool in tree" I
think is important,
> because if an alternative to TableGen is devised, then having its source
> within the LLVM tree ensures that it will be maintained and will develop
over
> time.
> 
> 	MartinO
> 
> -----Original Message-----
> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of Hal
> Finkel via llvm-dev
> Sent: 23 August 2017 19:45
> 
> >> There are also external tools that both produce and consume the
> >> TableGen sources.  Having a language that is *not* a
general-purpose
> >> programming language is a feature for these tools, not an
obstacle.
> >
> > I think that this goes both ways. For in-tree targets at least, we
> > want the source that is intended to be maintained by a human in tree.
> > So, the fact that an external tool can produce TableGen doesn't
really
> > solve the problem (unless we can also have that tool in tree, and
> > moreover, I don't want to encourage each backend to develop their
own
> > such tools independently).
> >
> >   -Hal
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.llvm.org_cgi-
> 2Dbin_mailman_listinfo_llvm-
> 2Ddev&d=DwIGaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=ELyOnT0WepII6UnFk-
> OSzxlGOXXSfAvOLT6E8iPwwJk&m=3CaXxl5qLdyIkph7nyWPmXGMujtWrfGy1
> vhu4HTioOE&s=4-n_marejkaKSUFZsrDX4E9l1P5iecvKUbpkvIlccpk&e=