llvm dev - Aug 2020 - TableGen trace facility

Yes, I understand the problem. To be more useful, TableGen would have to carry
the traces along with the classes and records and (re)display the values while
the substitutions are being made.

I'm writing a new Programmer's Guide for TableGen and have been digging
into the parse-time versus substitution-time issue. I haven't found a
document that makes it clear. Can you give a quick summary of the phases?

At 8/4/2020 05:01 PM, Nicolai Hähnle wrote:
>On Mon, Aug 3, 2020 at 6:04 PM Paul C. Anagnostopoulos via llvm-dev
><llvm-dev at lists.llvm.org> wrote:
>> A question for those of you who have developed complex TableGen files:
Do you think a trace facility would be useful during debugging? The idea is to
add a new statement to TableGen along these lines:
>>
>>   trace tag : value1, value2, ... ;
>>
>> When encountered, the TableGen parser would display the tag along with
the values of the specified value1, value2, etc. The tag is an identifier that
makes it easier to distinguish multiple traces.
>
>I'm not so sure that that's particularly useful or how it would even
>work. The issue is that at the point in time where the frontend parses
>those trace tags, most substitutions haven't taken place yet, so
>you'll usually get fairly trivial answers that by themselves aren't
>particularly helpful.
>
>Some form of inspection of how values are substituted would indeed be
>helpful, I just don't think the "trace" is quite it. In a
perfect
>world, we'd have some sort of "record database explorer" that
doesn't
>just let you look at all the final records (TableGen already allows
>you to do that), but also allows you to interactively explore their
>"history", i.e. how did the records come to be.
>
>Cheers,
>Nicolai
>
>-- 
>Lerne, wie die Welt wirklich ist,
>aber vergiss niemals, wie sie sein sollte.

On Tue, Aug 4, 2020 at 11:13 PM Paul C. Anagnostopoulos
<paul at windfall.com> wrote:
> Yes, I understand the problem. To be more useful, TableGen would have to
carry the traces along with the classes and records and (re)display the values
while the substitutions are being made.
>
> I'm writing a new Programmer's Guide for TableGen and have been
digging into the parse-time versus substitution-time issue. I haven't found
a document that makes it clear. Can you give a quick summary of the phases?
There aren't really any phases per se, in the sense that the TableGen
frontend doesn't have passes. Maybe that's a mistake, but that's how
it evolved.

So instead, every time a record is "instantiated" -- whether
that's by
defining a new class that derives from one or more pre-existing
classes, or by a def in a multi-class, or by a free-standing def, or a
defm (inside or outside a multiclass) -- any free variables that are
"resolved" as template parameters get their substitution applied. The
relevant let-statements take effect just after that.

However, and this is key, reference _between_ fields are only resolved
and substituted once the final defined record at the global scope is
produced. This is what makes:

class A<int x> {
  int a = x;
  int b = a + 1;
}

def B : A<5> { let a = 10; }

... result in b == 11 instead of b == 6.

There's probably subtlety here that I'm forgetting, but that's the
very short version of it ;)

Cheers,
Nicolai

>
> At 8/4/2020 05:01 PM, Nicolai Hähnle wrote:
> >On Mon, Aug 3, 2020 at 6:04 PM Paul C. Anagnostopoulos via llvm-dev
> ><llvm-dev at lists.llvm.org> wrote:
> >> A question for those of you who have developed complex TableGen
files: Do you think a trace facility would be useful during debugging? The idea
is to add a new statement to TableGen along these lines:
> >>
> >>   trace tag : value1, value2, ... ;
> >>
> >> When encountered, the TableGen parser would display the tag along
with the values of the specified value1, value2, etc. The tag is an identifier
that makes it easier to distinguish multiple traces.
> >
> >I'm not so sure that that's particularly useful or how it would
even
> >work. The issue is that at the point in time where the frontend parses
> >those trace tags, most substitutions haven't taken place yet, so
> >you'll usually get fairly trivial answers that by themselves
aren't
> >particularly helpful.
> >
> >Some form of inspection of how values are substituted would indeed be
> >helpful, I just don't think the "trace" is quite it. In a
perfect
> >world, we'd have some sort of "record database explorer"
that doesn't
> >just let you look at all the final records (TableGen already allows
> >you to do that), but also allows you to interactively explore their
> >"history", i.e. how did the records come to be.
> >
> >Cheers,
> >Nicolai
> >
> >--
> >Lerne, wie die Welt wirklich ist,
> >aber vergiss niemals, wie sie sein sollte.
>

-- 
Lerne, wie die Welt wirklich ist,
aber vergiss niemals, wie sie sein sollte.

Are all the records collected as they are parsed, with template parameter
substitution and lets, and *then*, after all records are collected, a
"pass" is made to calculate the inter-field expressions?

Once I understand this, I will add a section to the new guide to explain it. I
presume it is the case that this behavior should be publicized.

It also appears to be the case that a record is created and bound to its name
before fields are inherited from its superclasses, which is why you can write:

class A <dag d> {
  dag the_dag = d;
}

def rec1 : A<(ops rec1)>

Do I understand that correctly?

At 8/4/2020 05:38 PM, Nicolai Hähnle wrote:
>On Tue, Aug 4, 2020 at 11:13 PM Paul C. Anagnostopoulos
><paul at windfall.com> wrote:
>> Yes, I understand the problem. To be more useful, TableGen would have
to carry the traces along with the classes and records and (re)display the
values while the substitutions are being made.
>>
>> I'm writing a new Programmer's Guide for TableGen and have been
digging into the parse-time versus substitution-time issue. I haven't found
a document that makes it clear. Can you give a quick summary of the phases?
>
>There aren't really any phases per se, in the sense that the TableGen
>frontend doesn't have passes. Maybe that's a mistake, but that's
how
>it evolved.
>
>So instead, every time a record is "instantiated" -- whether
that's by
>defining a new class that derives from one or more pre-existing
>classes, or by a def in a multi-class, or by a free-standing def, or a
>defm (inside or outside a multiclass) -- any free variables that are
>"resolved" as template parameters get their substitution applied.
The
>relevant let-statements take effect just after that.
>
>However, and this is key, reference _between_ fields are only resolved
>and substituted once the final defined record at the global scope is
>produced. This is what makes:
>
>class A<int x> {
>  int a = x;
>  int b = a + 1;
>}
>
>def B : A<5> { let a = 10; }
>
>... result in b == 11 instead of b == 6.
>
>There's probably subtlety here that I'm forgetting, but that's
the
>very short version of it ;)
>
>Cheers,
>Nicolai
>
>
>>
>> At 8/4/2020 05:01 PM, Nicolai Hähnle wrote:
>> >On Mon, Aug 3, 2020 at 6:04 PM Paul C. Anagnostopoulos via llvm-dev
>> ><llvm-dev at lists.llvm.org> wrote:
>> >> A question for those of you who have developed complex
TableGen files: Do you think a trace facility would be useful during debugging?
The idea is to add a new statement to TableGen along these lines:
>> >>
>> >>   trace tag : value1, value2, ... ;
>> >>
>> >> When encountered, the TableGen parser would display the tag
along with the values of the specified value1, value2, etc. The tag is an
identifier that makes it easier to distinguish multiple traces.
>> >
>> >I'm not so sure that that's particularly useful or how it
would even
>> >work. The issue is that at the point in time where the frontend
parses
>> >those trace tags, most substitutions haven't taken place yet,
so
>> >you'll usually get fairly trivial answers that by themselves
aren't
>> >particularly helpful.
>> >
>> >Some form of inspection of how values are substituted would indeed
be
>> >helpful, I just don't think the "trace" is quite it.
In a perfect
>> >world, we'd have some sort of "record database
explorer" that doesn't
>> >just let you look at all the final records (TableGen already allows
>> >you to do that), but also allows you to interactively explore their
>> >"history", i.e. how did the records come to be.
>> >
>> >Cheers,
>> >Nicolai
>> >
>> >--
>> >Lerne, wie die Welt wirklich ist,
>> >aber vergiss niemals, wie sie sein sollte.

I wasn't sure how to respond to John Byrd's post, since it wasn't
addressed to me. So I've responded to Nicolai's.

I'm reasonably far along in the process of writing a new Programmer's
Guide for TableGen. I will continue working on it and submit it for review. I
expect to do some rewriting as a result.

John: Would you like me to respect a copyright on your documents linked below? I
wouldn't take any text verbatim, but some of the ways of describing TableGen
give me ideas.

----------------------------------------
Similarly, I've just survived writing most of a new backend, and
TableGen's
documentation was unfortunate.

Chris's old original documentation was incomplete, but it was clear, and as
such the original sketches were superior to the current state of TableGen
documentation.

TableGen needs a proper programmer's guide, which differs from a
reference.  I actually started such a rewrite here, but I never tried
submitting it to Phabricator, because it wasn't clear to me who was
responsible for review:

https://github.com/johnwbyrd/llvm-mos/blob/master/llvm/docs/TableGen/index.rst

https://github.com/johnwbyrd/llvm-mos/blob/master/llvm/docs/TableGen/Deficiencies.rst


Most of the concepts behind TableGen are simple.  They should be
communicated simply, as per Chris's original documentation.  The things
that are not simple, were developed as special-purpose solutions to
practical problems.

The current documentation should be rewritten almost entirely from scratch,
with a strong focus on organizing the document in a reasonable reading
order, with a target audience of those who are working on LLVM backends.
Patching the documentation has gotten it to its current state of confusion,
and the process should be discontinued.

Likewise, I'll mention that if anyone wants to finance authorship of proper
TableGen and/or backend documentation, I'd be game to research it and write
it.
---

John Byrd