llvm dev - Apr 2016 - [cfe-dev] [PATCH/DRAFT] Embed metadata into object file

> On Apr 1, 2016, at 5:20 AM, Christian Dietrich via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Adrian Prantl <aprantl at apple.com> writes:
> 
>> Depending on your needs, just using a global with the “section”
>> attribute might also work for you:
>> http://llvm.org/docs/LangRef.html#global-variables
> 
> I was aware of that possibility. But, there are several drawbacks to
> using global variables and the current infrastructure from a clang
> plugin's point of view:
> 
> 1. It feels like a hack. It is not an idiomatic way of transporting
>   information alongside with an translation unit.
> 
> 2. A clang plugin is mostly defined by it's ASTConsumer; there is no
>   direct access to the produced LLVM intermediate representation. I
>   would have to insert AST elements that result in the attributed
>   global variable. [1]
> 
> 3. A more idealistic problem I have with this solution is that we change
>   the actual (semantic) content of the module. But I only want to carry
>   _metadata_ about the module alongside.
> 
> 4. There already is an good metadata infrastructure in LLVM and a very
>   good attribute infrastructure in clang. Why not utilize them?
> 
> So my question is:
> 
>   I want to generate and attach (structured) metadata at every point in
>   life cycle (AST, LLVM module, object file) of a module and retrieve it
>   from the compilation results. How can I do that?
Worded like that, I can see a close analogy with  "Debug Info".

> At the moment, my patches are only a proposal to solve this for a very
> specifc use case (mine): attaching single strings. I think, furthermore,
> that it would also be handy to materialize and retrieve more complex
> metadata types to/from object files. For example, we could serialize
> LLVM metadata trees into separate sections:
> 
>   !llvm.extra_sections = {!0}
>   !0 = !{!"clang.myanalysis", !1, !2}
>   !1 = !{ i32 15, i32 28, i32 142}
>   !2 = !{!"key", i32 10000, !"key2", i32 9999}
> 
> I'm not sure whether this would be useful and reusable for others. So:
> Do you think this would be useful for other developers as well?
> 
> chris
> 
> 
> [1] This is a _seperate_ drawback of clang plugins at the moment: They
>    cannot define an LLVM IR transformation that should be applied when
>    the plugin is active.
You can write clang plugin that are LLVM pass and insert them in the pipeline.

> I think it would be useful for many analyzes
>    to have access both to the AST and to the IR.
This is more fuzzy to me, I don't know enough about clang but I'm not
sure the design allow to keep a link from the IR to the clang AST? (If it is the
case, I'd be curious to see how it works).

Best,

-- 
Mehdi

Mehdi Amini <mehdi.amini at apple.com> writes:
> Worded like that, I can see a close analogy with  "Debug Info".
Indeed, it is very similar, but there are some differences and
shortcoming, if a developer only wants to smuggle some metadata out in a
very specific format:

For the IR->ELF path, the debug information is encoded as Dwarf (or
  something else) in the binary. The plugin developer has not much
  control about the binary format of the data. !llvm.extra_sections
  would give a quite fine-grained control.

For the AST->IR path, I don't see an easy way to annotate a few pieces
  of information in the AST without spinning up the whole
  debug-information generation process.
>> [1] This is a _seperate_ drawback of clang plugins at the moment: They
>>    cannot define an LLVM IR transformation that should be applied when
>>    the plugin is active.
>
> You can write clang plugin that are LLVM pass and insert them in the
> pipeline.
I can? How? I have not seen any possibility to inject a LLVM pass into
the Clang CodeGen PassManager infrastructure from a clang plugin, which
is also FrontentAction like CodeGen.
>
>> I think it would be useful for many analyzes
>>    to have access both to the AST and to the IR.
>
> This is more fuzzy to me, I don't know enough about clang but I'm
not
> sure the design allow to keep a link from the IR to the clang AST? (If
> it is the case, I'd be curious to see how it works).
If you restrict the possible subjects to top-level functions, global
variables, and the compilation unit, this should be implementable quite
straight forward.

chris
-- 
Christian Dietrich, M.Sc. (Wissenschaftlicher Mitarbeiter)
Lehrstuhl für Informatik 4 (Verteilte Systeme und Betriebssysteme)
Friedrich-Alexander-Universität Erlangen-Nürnberg
Martensstr. 1
91058 Erlangen

Tel:    (09131) 85-27280
Fax:    (09131) 85-28732
eMail:  christian.dietrich at fau.de
WWW:    http://www4.cs.fau.de/~dietrich

> On Apr 5, 2016, at 7:51 AM, Christian Dietrich <dietrich at
cs.fau.de> wrote:
> 
> Mehdi Amini <mehdi.amini at apple.com> writes:
> 
>> Worded like that, I can see a close analogy with  "Debug
Info".
> 
> Indeed, it is very similar, but there are some differences and
> shortcoming, if a developer only wants to smuggle some metadata out in a
> very specific format:
> 
> For the IR->ELF path, the debug information is encoded as Dwarf (or
>  something else) in the binary. The plugin developer has not much
>  control about the binary format of the data. !llvm.extra_sections
>  would give a quite fine-grained control.
> 
> For the AST->IR path, I don't see an easy way to annotate a few
pieces
>  of information in the AST without spinning up the whole
>  debug-information generation process.

Yes, I meant conceptually it is the same as debug info, so it makes sense to
solve it the same way (i.e. metadata + backend support for codegen).
> 
>>> [1] This is a _seperate_ drawback of clang plugins at the moment:
They
>>>   cannot define an LLVM IR transformation that should be applied
when
>>>   the plugin is active.
>> 
>> You can write clang plugin that are LLVM pass and insert them in the
>> pipeline.
> 
> I can? How? I have not seen any possibility to inject a LLVM pass into
> the Clang CodeGen PassManager infrastructure from a clang plugin, which
> is also FrontentAction like CodeGen.
The FrontendAction is totally separated from the LLVM-pass, but I don’t see why
your plugin can’t contain both.

To load an LLVM pass into the pipeline, see what Polly is doing for instance:
http://polly.llvm.org/example_load_Polly_into_clang.html

— 
Mehdi


> 
>> 
>>> I think it would be useful for many analyzes
>>>   to have access both to the AST and to the IR.
>> 
>> This is more fuzzy to me, I don't know enough about clang but
I'm not
>> sure the design allow to keep a link from the IR to the clang AST? (If
>> it is the case, I'd be curious to see how it works).
> 
> If you restrict the possible subjects to top-level functions, global
> variables, and the compilation unit, this should be implementable quite
> straight forward.
> 
> chris
> -- 
> Christian Dietrich, M.Sc. (Wissenschaftlicher Mitarbeiter)
> Lehrstuhl für Informatik 4 (Verteilte Systeme und Betriebssysteme)
> Friedrich-Alexander-Universität Erlangen-Nürnberg
> Martensstr. 1
> 91058 Erlangen
> 
> Tel:    (09131) 85-27280
> Fax:    (09131) 85-28732
> eMail:  christian.dietrich at fau.de
> WWW:    http://www4.cs.fau.de/~dietrich