llvm dev - Mar 2016 - [PATCH/DRAFT] Embed metadata into object file

Hi,

so this is my first contribution to LLVM/clang, so I hope I come close
to the required coding standards and guidelines.

First, I will describe the scenario I want to solve: For a few days, the
clang plugin interface allows to execute the a plugin just before the
actual main action (e.g., compiling an translation unit). In my case,
the plugin we're developing will analyze the AST and generate some
information. This information should be available in the generated
object file. So, I had to solve the question: how do I smuggle some
metadata on the module level, from the frontend to the generated object
file (in an hopefully sane and reusable manner).

So, the solution I thought about was the following. The gathered
information is but into a NamedMetadataStringAttr and attached to the
TranslationUnitDecl. This would be, as far as I can see, the first
annotation on the TranslationUnit level. Attributes seem to me the best
solution at that point, since there is already a good infrastructure in
clang.

   TranslationUnitDecl
   | NamedMetadataAttr implicit llvm.extra_section clang.analysis
"Information"

The clang CodeGen then generates a named metadata node on the
llvm::Module level:

   !llvm.extra_sections = {!0}
   !0 = !{!"clang.analysis", !"Information"}

The attached patch, then takes all key-value pairs from the named
llvm.extra_sections named metadata, and appends them as zero-terminated
strings to the desired section.

I'm not sure, wheter this is the best solution to the problem, and
wheter it is general enough. I was also thinking about the possibility
to attach LLVM passes from the plugin interface to the clang/CodeGen
backend, but I could not figure out how to do this without breaking all
the used abstraction.

I post this to llvm-dev, as well as, to cfe-dev, since both changes,
although idependend of each other, relate to one another.

chris
-- 
Christian Dietrich, M.Sc. (Wissenschaftlicher Mitarbeiter)
Lehrstuhl für Informatik 4 (Verteilte Systeme und Betriebssysteme)
Friedrich-Alexander-Universität Erlangen-Nürnberg
Martensstr. 1
91058 Erlangen

Tel:    (09131) 85-27280
Fax:    (09131) 85-28732
eMail:  christian.dietrich at fau.de
WWW:    http://www4.cs.fau.de/~dietrich
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Attach-LLVM-metadata-with-NamedMetadataStringAttr.patch
Type: text/x-diff
Size: 3422 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160330/e860a8fb/attachment.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Use-llvm.extra_section-to-smuggle-data-into-object-s.patch
Type: text/x-diff
Size: 9411 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160330/e860a8fb/attachment-0001.patch>

> On Mar 30, 2016, at 6:47 AM, Christian Dietrich via cfe-dev <cfe-dev at
lists.llvm.org> wrote:
> 
> Hi,
> 
> so this is my first contribution to LLVM/clang, so I hope I come close
> to the required coding standards and guidelines.
> 
> First, I will describe the scenario I want to solve: For a few days, the
> clang plugin interface allows to execute the a plugin just before the
> actual main action (e.g., compiling an translation unit). In my case,
> the plugin we're developing will analyze the AST and generate some
> information. This information should be available in the generated
> object file. So, I had to solve the question: how do I smuggle some
> metadata on the module level, from the frontend to the generated object
> file (in an hopefully sane and reusable manner).
> 
> So, the solution I thought about was the following. The gathered
> information is but into a NamedMetadataStringAttr and attached to the
> TranslationUnitDecl. This would be, as far as I can see, the first
> annotation on the TranslationUnit level. Attributes seem to me the best
> solution at that point, since there is already a good infrastructure in
> clang.
> 
>   TranslationUnitDecl
>   | NamedMetadataAttr implicit llvm.extra_section clang.analysis
"Information"
> 
> The clang CodeGen then generates a named metadata node on the
> llvm::Module level:
> 
>   !llvm.extra_sections = {!0}
>   !0 = !{!"clang.analysis", !"Information"}
> 
> The attached patch, then takes all key-value pairs from the named
> llvm.extra_sections named metadata, and appends them as zero-terminated
> strings to the desired section.
Depending on your needs, just using a global with the “section” attribute might
also work for you:
http://llvm.org/docs/LangRef.html#global-variables


-- adrian
> 
> I'm not sure, wheter this is the best solution to the problem, and
> wheter it is general enough. I was also thinking about the possibility
> to attach LLVM passes from the plugin interface to the clang/CodeGen
> backend, but I could not figure out how to do this without breaking all
> the used abstraction.
> 
> I post this to llvm-dev, as well as, to cfe-dev, since both changes,
> although idependend of each other, relate to one another.
> 
> chris
> -- 
> Christian Dietrich, M.Sc. (Wissenschaftlicher Mitarbeiter)
> Lehrstuhl für Informatik 4 (Verteilte Systeme und Betriebssysteme)
> Friedrich-Alexander-Universität Erlangen-Nürnberg
> Martensstr. 1
> 91058 Erlangen
> 
> Tel:    (09131) 85-27280
> Fax:    (09131) 85-28732
> eMail:  christian.dietrich at fau.de
> WWW:    http://www4.cs.fau.de/~dietrich
>
<0001-Attach-LLVM-metadata-with-NamedMetadataStringAttr.patch><0001-Use-llvm.extra_section-to-smuggle-data-into-object-s.patch>_______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

Adrian Prantl <aprantl at apple.com> writes:
> Depending on your needs, just using a global with the “section”
> attribute might also work for you:
> http://llvm.org/docs/LangRef.html#global-variables
I was aware of that possibility. But, there are several drawbacks to
using global variables and the current infrastructure from a clang
plugin's point of view:

1. It feels like a hack. It is not an idiomatic way of transporting
   information alongside with an translation unit.

2. A clang plugin is mostly defined by it's ASTConsumer; there is no
   direct access to the produced LLVM intermediate representation. I
   would have to insert AST elements that result in the attributed
   global variable. [1]

3. A more idealistic problem I have with this solution is that we change
   the actual (semantic) content of the module. But I only want to carry
   _metadata_ about the module alongside.

4. There already is an good metadata infrastructure in LLVM and a very
   good attribute infrastructure in clang. Why not utilize them?

So my question is:

   I want to generate and attach (structured) metadata at every point in
   life cycle (AST, LLVM module, object file) of a module and retrieve it
   from the compilation results. How can I do that?

At the moment, my patches are only a proposal to solve this for a very
specifc use case (mine): attaching single strings. I think, furthermore,
that it would also be handy to materialize and retrieve more complex
metadata types to/from object files. For example, we could serialize
LLVM metadata trees into separate sections:

   !llvm.extra_sections = {!0}
   !0 = !{!"clang.myanalysis", !1, !2}
   !1 = !{ i32 15, i32 28, i32 142}
   !2 = !{!"key", i32 10000, !"key2", i32 9999}

I'm not sure whether this would be useful and reusable for others. So:
Do you think this would be useful for other developers as well?

chris


[1] This is a _seperate_ drawback of clang plugins at the moment: They
    cannot define an LLVM IR transformation that should be applied when
    the plugin is active. I think it would be useful for many analyzes
    to have access both to the AST and to the IR.
-- 
Christian Dietrich, M.Sc. (Wissenschaftlicher Mitarbeiter)
Lehrstuhl für Informatik 4 (Verteilte Systeme und Betriebssysteme)
Friedrich-Alexander-Universität Erlangen-Nürnberg
Martensstr. 1
91058 Erlangen

Tel:    (09131) 85-27280
Fax:    (09131) 85-28732
eMail:  christian.dietrich at fau.de
WWW:    http://www4.cs.fau.de/~dietrich