llvm dev - Mar 2015 - [LLVMdev] TargetParser - Always build all table-gen files?

Hi Folks,

I'm back looking at the target parser that we discussed last year, and
I have some questions.

http://lists.cs.uiuc.edu/pipermail/cfe-dev/2014-August/038699.html

There is no way I can create a target-independent parser in lib/Target
without re-encoding all the information about the targets that we
already have in the table-gen files on all targets' directories. Doing
that would be folly.

But building all targets by default goes against the current thinking,
which I quite welcome and think we should not break it. We need to
find a common ground.

My idea is to build all table-gen files on all targets, but not build
all targets' files and not include them in the final libraries /
binaries. Since the table-gen'ed files don't end up there by
themselves, only what we use from them (in the target parser) will be
included in the final binaries.

Does that sound like a good idea? If so, how would one go about doing
that? I'm guessing it's a change solely in CMake files, and could be
done independently of the target parser implementation, right?

cheers,
--renato

On 9 March 2015 at 12:02, Renato Golin <renato.golin at linaro.org>
wrote:
> My idea is to build all table-gen files on all targets, but not build
> all targets' files and not include them in the final libraries /
> binaries. Since the table-gen'ed files don't end up there by
> themselves, only what we use from them (in the target parser) will be
> included in the final binaries.
Attached is a patch that kind of does that, but not completely. CMakes
finishes ok, but make can't find the td files.

Basically, I moved the tablegen() registration process to a separate
file (CMakeTblGen.txt), which is included from CMakeLists.txt in each
target directory and also in the root lib/Target/CMakeLists.txt, so
that I can NOT include all targets while including ALL table-gen files
that weren't included by the targets_to_build.

The problem is that, when I include it from lib/Target/CMakeLists.txt,
the tablegen(LLVM File.td -opts) makes File.td relative to its own dir
(lib/Target/File.td) and not the targets'.

The only way I can think of solving this is to change how tablegen()
handles the file name, with some "if(IS_ABSOLUTE)", but looking
further, the file name is used to more than just a filename, and gets
included in some variables names, etc. Also, I can't add another
option to the end of tablegen() because, AFAICS, this function is
variadic, with all table-gen options following the file name...

Ideas?

--renato
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cmake_tblgen_parser.patch
Type: text/x-patch
Size: 22743 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150309/f40985e5/attachment.bin>

To be clear, TargetParser is about parsing subtarget CPUs and features,
right?

I'm not very familiar with how the current system works, but it would be a
shame to run tablegen for all targets when the user just wants one backend.

The subtarget feature flags aren't very much data. If I were doing this
today, I would make one subtarget info library and put all the subtarget
features in there. The subtarget information is far less than the
instruction table information. For example:
lib/Target/Subtargets/X86Subtargets.td
lib/Target/Subtargets/ARMSubtargets.td
...

Now that we have a lot of backends and lots of out of tree backends, this
might be too disruptive. The answer might be that we have to live with
duplicating the subtarget features in clang and building some kind of
sanity checking mechanism to keep them in sync.

On Mon, Mar 9, 2015 at 5:02 AM, Renato Golin <renato.golin at linaro.org>
wrote:
> Hi Folks,
>
> I'm back looking at the target parser that we discussed last year, and
> I have some questions.
>
> http://lists.cs.uiuc.edu/pipermail/cfe-dev/2014-August/038699.html
>
> There is no way I can create a target-independent parser in lib/Target
> without re-encoding all the information about the targets that we
> already have in the table-gen files on all targets' directories. Doing
> that would be folly.
>
> But building all targets by default goes against the current thinking,
> which I quite welcome and think we should not break it. We need to
> find a common ground.
>
> My idea is to build all table-gen files on all targets, but not build
> all targets' files and not include them in the final libraries /
> binaries. Since the table-gen'ed files don't end up there by
> themselves, only what we use from them (in the target parser) will be
> included in the final binaries.
>
> Does that sound like a good idea? If so, how would one go about doing
> that? I'm guessing it's a change solely in CMake files, and could
be
> done independently of the target parser implementation, right?
>
> cheers,
> --renato
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150309/9575e264/attachment.html>

On 9 March 2015 at 18:21, Reid Kleckner <rnk at google.com>
wrote:
> To be clear, TargetParser is about parsing subtarget CPUs and features,
> right?
In the first stage, yes. But there's a lot more. I hope this ends up
being a much larger infrastructure to query for target support, not
just parsing strings (which we can cope with duplication), but
defining architectural behaviour, which is a whole different case.

Clang is only one user. Ever other front-end has to do the same. All
other tools (lli, llc, etc) have to duplicate string parsing, and any
out-of-tree plugin will have to do the same, if they need anything
that is not built as a back-end.

There is a lot in Clang's driver that could be greatly simplified if
we had a proper unified target description.

> I'm not very familiar with how the current system works, but it would
be a
> shame to run tablegen for all targets when the user just wants one backend.
Why? The only effect of this is to process all targets' table-gen
files at build time, mostly once for the targets you rarely use. They
will not be included in any library, binary or archive if you don't
use the specific back-end, and the TargetParser will only use the
necessary tables to get target information.

> The subtarget feature flags aren't very much data. If I were doing this
> today, I would make one subtarget info library and put all the subtarget
> features in there. The subtarget information is far less than the
> instruction table information. For example:
> lib/Target/Subtargets/X86Subtargets.td
> lib/Target/Subtargets/ARMSubtargets.td
It's not just CPU names. For instance, Clang has support for register
names. Right now, they're all accepted (x86 names for ARM code)
because Clang doesn't know better. Another example is assembler /
linker flags (-Wa,-Wl), which are not supported at all for the
integrated assembler and would be another user of parsing and
inspecting capabilities.

All the information is there, in the table-gen files, and could be
easily accessed into an abstraction layer (TargetDescription or
whatever), which TargetParser could use, as well as all other tools,
including lld, lldb, etc. We're not talking about duplication, we're
talking about multiplication by a factor a lot more than two.

cheers,
--renato

On 9 March 2015 at 17:40, Renato Golin <renato.golin at linaro.org>
wrote:
> The only way I can think of solving this is to change how tablegen()
> handles the file name, with some "if(IS_ABSOLUTE)", but looking
> further, the file name is used to more than just a filename, and gets
> included in some variables names, etc. Also, I can't add another
> option to the end of tablegen() because, AFAICS, this function is
> variadic, with all table-gen options following the file name...
CMake turned out a lot more powerful than I was imagining, and the
change was really simple.

Let's follow it here:

http://reviews.llvm.org/D8195

The biggest question now is: do we have a consensus that building all
table-gen files is worth the massive de-duplication of current code
and future work that will follow?

cheers,
--renato