llvm dev - Sep 2020 - Making library calls for obj2yaml functionalities

Hi All,

Following up on
https://lists.llvm.org/pipermail/llvm-dev/2020-July/143512.html, and
https://reviews.llvm.org/D85408, we would like to consider a design which
allows external tools to read the structured contents of the .bb_addr_map
section with library calls into an LLVM library. At the same time, we need
to have tools/obj2yaml tests in place for bb_addr_map. So it sounds like
the perfect place to do it. However, the current structure does not expose
the obj2yaml functionalities under lib/ObjectYAML.

In general, there seems to be an inconsistency regarding how obj2yaml and
yaml2obj tools are structured. The latter has a nice wrapper which calls
environment-dependent functions under lib/ObjectYAML, but the former has
environment-dependent source files under tools/obj2yaml (elf2yaml.cc,
coff2yaml.cc, etc).

I wanted to reach out to ask if there is any alternative or get an idea
about the amount of refactoring work that is required to make the structure
friendlier.

bests,
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200916/fb080ec6/attachment.html>

Hi Rahman,

Traditionally, the ability to read sections is a feature added to
llvm-readobj/llvm-readelf. For some sections, it delegates to methods in
places like the Object library and BinaryFormat, but for the more
specialised sections, it typically has code local to itself doing the work.
The same is true for other dumping tools like obj2yaml and llvm-objdump,
which means in some cases, we have multiple varieties of parsers for the
same thing. I'm not sure there's necessarily a strong motivation for
doing
so, however, so I'd be happy to support functionality being added elsewhere
in one of those libraries, which tools like obj2yaml and llvm-readobj can
hook into. I'm also happy to support refactoring that improves code reuse
within the tools, though I don't have any further ideas on this.

Can I ask what your motivation for using obj2yaml is in this context? If
it's just for testing purposes, adding support to llvm-readobj/llvm-readelf
would be the more normal way, as it allows you to dump just that section.

Only tangentially relatedly, I've only just seen your previous patch/email
thread, and I do have one thing I'd like to ask if it can be changed. At
the moment, the section type is SHT_PROGBITS, but I think it would be
better, if possible, to define a new SHT_* type for the new section? In
general, it is bad design to rely on section names to distinguish between
different kinds of sections - this requires the linker and other tools to
have to do unnecessary string comparisons, which are slower and messier
than switching on the sh_type field.

James

On Wed, 16 Sep 2020 at 22:00, Rahman Lavaee <rahmanl at google.com> wrote:
> Hi All,
>
> Following up on
> https://lists.llvm.org/pipermail/llvm-dev/2020-July/143512.html, and
> https://reviews.llvm.org/D85408, we would like to consider a design which
> allows external tools to read the structured contents of the .bb_addr_map
> section with library calls into an LLVM library. At the same time, we need
> to have tools/obj2yaml tests in place for bb_addr_map. So it sounds like
> the perfect place to do it. However, the current structure does not expose
> the obj2yaml functionalities under lib/ObjectYAML.
>
> In general, there seems to be an inconsistency regarding how obj2yaml and
> yaml2obj tools are structured. The latter has a nice wrapper which calls
> environment-dependent functions under lib/ObjectYAML, but the former has
> environment-dependent source files under tools/obj2yaml (elf2yaml.cc,
> coff2yaml.cc, etc).
>
> I wanted to reach out to ask if there is any alternative or get an idea
> about the amount of refactoring work that is required to make the structure
> friendlier.
>
> bests,
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200917/3d958ec0/attachment.html>

James,
Thanks for the detailed response. Please see my thoughts inline.

On Thu, Sep 17, 2020 at 12:33 AM James Henderson <
jh7370.2008 at my.bristol.ac.uk> wrote:
> Hi Rahman,
>
> Traditionally, the ability to read sections is a feature added to
> llvm-readobj/llvm-readelf. For some sections, it delegates to methods in
> places like the Object library and BinaryFormat, but for the more
> specialised sections, it typically has code local to itself doing the work.
> The same is true for other dumping tools like obj2yaml and llvm-objdump,
> which means in some cases, we have multiple varieties of parsers for the
> same thing. I'm not sure there's necessarily a strong motivation
for doing
> so, however, so I'd be happy to support functionality being added
elsewhere
> in one of those libraries, which tools like obj2yaml and llvm-readobj can
> hook into. I'm also happy to support refactoring that improves code
reuse
> within the tools, though I don't have any further ideas on this.
>
> Can I ask what your motivation for using obj2yaml is in this context? If
> it's just for testing purposes, adding support to
llvm-readobj/llvm-readelf
> would be the more normal way, as it allows you to dump just that section.
>
Other than testing, we currently have code in an external tool called
create_llvm_prof for parsing the ".bb_addr_map" section (+Han Shen
<shenhan at google.com> who's the main developer of that tool) and
loading it
in memory. It would've been great if we could just link with an LLVM
library which includes the data-structure and parsing support. It looks
like llvm-readelf is more structured around dumping/printing. However, I
can see structures like ELFYAML::StackSizesSection in
lib/ObjectYAML/ELFYAML.h which could be simply passed around.

Only tangentially relatedly, I've only just seen your previous
patch/email
> thread, and I do have one thing I'd like to ask if it can be changed.
At
> the moment, the section type is SHT_PROGBITS, but I think it would be
> better, if possible, to define a new SHT_* type for the new section? In
> general, it is bad design to rely on section names to distinguish between
> different kinds of sections - this requires the linker and other tools to
> have to do unnecessary string comparisons, which are slower and messier
> than switching on the sh_type field.
>
Yes, having a specific ELF section type would be great. Though I am
clueless about how this change will be received given that the
".stack_sizes" section also uses SHT_PROGBITS. I can definitely look
into
it if I get some assurance.

James
>
> On Wed, 16 Sep 2020 at 22:00, Rahman Lavaee <rahmanl at google.com>
wrote:
>
>> Hi All,
>>
>> Following up on
>> https://lists.llvm.org/pipermail/llvm-dev/2020-July/143512.html, and
>> https://reviews.llvm.org/D85408, we would like to consider a design
>> which allows external tools to read the structured contents of the
>> .bb_addr_map section with library calls into an LLVM library. At the
same
>> time, we need to have tools/obj2yaml tests in place for bb_addr_map. So
it
>> sounds like the perfect place to do it. However, the current structure
does
>> not expose the obj2yaml functionalities under lib/ObjectYAML.
>>
>> In general, there seems to be an inconsistency regarding how obj2yaml
and
>> yaml2obj tools are structured. The latter has a nice wrapper which
calls
>> environment-dependent functions under lib/ObjectYAML, but the former
has
>> environment-dependent source files under tools/obj2yaml (elf2yaml.cc,
>> coff2yaml.cc, etc).
>>
>> I wanted to reach out to ask if there is any alternative or get an idea
>> about the amount of refactoring work that is required to make the
structure
>> friendlier.
>>
>> bests,
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200917/1860e600/attachment.html>