llvm dev - Mar 2019 - [RFC] Upstream ObjCMetadata Reader Library

Overview
======We are planning to open source our internal implementation of Objective-C
metadata reader library to LLVM. Unlike the C style objective-c metadata reader
in llvm-objdump, it is designed to be a reusable C++ library to parse
objective-c metadata from binaries with LLVM style error handling. The current
implementation has an interface that resembles libObject in and it supports
parsing objective-c metadata from MachO object file or LLVM bitcode that is
generated by clang/swift compiler. It has the flexibility to be extended to
support other objective-c metadata format if needed.
You can find an old open-source version of implementation here[1]. The version
we planned to open source has the same design but it will come with some bug
fixes and improvements, and it will be rebased to the trunk.

Background
========We think it is very beneficial to have an objective-c metadata library
in LLVM. We use the library in many tools, including tapi which we are currently
in the process of open sourcing. Some have been asking for this libraries on the
list or off the list to use it in the contexts like JIT[2] or macports.

Originally, the open source of this library is delayed because we thought it
would be good if we can achieve following two goals before upstreaming:
An improved libObject interface: we are expecting libObject interfaces to be
changed which ObjCMetadata library should be updated to match.
Replace the parser inside llvm-objdump.
The first one has been talked for years but never put into action. The second
one requires a strict output match because llvm-objdump is a replacement for
otool-classic on darwin platform, and doing so will require LLVM infrastructure
improvements on how to handle errors. With some discussion, we believe both
tasks can be done in trunk when it is time, rather than waiting for all the
requirements are met.

Design
====ObjCMetadata library comes with headers (in
llvm/include/llvm/ObjCMetadata)[3], implementations (in
llvm/lib/ObjCMetadata)[4] and we will also implement a tool similar to a
stripped down version of [5] for testing.
Here is an example how to use the library by extracting and printing the objc
class names:
```
  MachOMetadata ObjCInfo(InputObject);
  if (auto ObjCClasses = ObjCInfo.classes()) {
    for (auto c : *ObjCClasses) {
      auto ObjCClass = *c;
      if (!ObjCClass) {
        handleError(ObjCClass.takeError());
        continue;
      }
      auto name = ObjCClass->getName();
      if (!name) {
        handleError(name.takeError());
        continue;
      }
      outs() << *name << "\n";
  }
```

I am preparing a patch and will send out when it is ready. Let me know if you
have any questions or feedbacks.

Thanks

Steven

[1] https://opensource.apple.com/source/clang/clang-800.0.38
[2] http://lists.llvm.org/pipermail/llvm-dev/2016-November/106995.html
[3]
https://opensource.apple.com/source/clang/clang-800.0.38/src/lib/ObjCMetadata/
[4]
https://opensource.apple.com/source/clang/clang-800.0.38/src/include/llvm/ObjCMetadata/
[5]
https://opensource.apple.com/source/clang/clang-800.0.38/src/tools/api-analyzer/

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190322/98e1c570/attachment-0001.html>

Hi Steven,

This sounds like a very useful tool.  In Clang, we've been very careful 
to maintain an abstraction layer over different Objective-C runtime 
implementations.  Your email talked only about the Apple runtime - do 
you already have an abstraction layer for the runtime-specific details, 
or is this something that will need to be added prior to upstreaming?

David

On 22/03/2019 18:34, Steven Wu via llvm-dev wrote:
> Overview
> ======> We are planning to open source our internal implementation of 
> Objective-C metadata reader library to LLVM. Unlike the C style 
> objective-c metadata reader in llvm-objdump, it is designed to be a 
> reusable C++ library to parse objective-c metadata from binaries with 
> LLVM style error handling. The current implementation has an interface 
> that resembles libObject in and it supports parsing objective-c metadata 
> from MachO object file or LLVM bitcode that is generated by clang/swift 
> compiler. It has the flexibility to be extended to support other 
> objective-c metadata format if needed.
> You can find an old open-source version of implementation here[1]. The 
> version we planned to open source has the same design but it will come 
> with some bug fixes and improvements, and it will be rebased to the trunk.
> 
> Background
> ========> We think it is very beneficial to have an objective-c metadata
library
> in LLVM. We use the library in many tools, including tapi which we are 
> currently in the process of open sourcing. Some have been asking for 
> this libraries on the list or off the list to use it in the contexts 
> like JIT[2] or macports.
> 
> Originally, the open source of this library is delayed because we 
> thought it would be good if we can achieve following two goals before 
> upstreaming:
> 
>   * An improved libObject interface: we are expecting libObject
>     interfaces to be changed which ObjCMetadata library should be
>     updated to match.
>   * Replace the parser inside llvm-objdump.
> 
> The first one has been talked for years but never put into action. The 
> second one requires a strict output match because llvm-objdump is a 
> replacement for otool-classic on darwin platform, and doing so will 
> require LLVM infrastructure improvements on how to handle errors. With 
> some discussion, we believe both tasks can be done in trunk when it is 
> time, rather than waiting for all the requirements are met.
> 
> Design
> ====> ObjCMetadata library comes with headers (in 
> llvm/include/llvm/ObjCMetadata)[3], implementations (in 
> llvm/lib/ObjCMetadata)[4] and we will also implement a tool similar to a 
> stripped down version of [5] for testing.
> Here is an example how to use the library by extracting and printing the 
> objc class names:
> ```
>    MachOMetadata ObjCInfo(InputObject);
> *if* (*auto* ObjCClasses = ObjCInfo.classes()) {
> *for* (*auto* c : *ObjCClasses) {
> *auto* ObjCClass = *c;
> *if* (!ObjCClass) {
>          handleError(ObjCClass.takeError());
> *continue*;
>        }
> *auto* name = ObjCClass->getName();
> *if* (!name) {
>          handleError(name.takeError());
> *continue*;
>        }
>        outs() << *name << "\n";
>    }
> ```
> 
> I am preparing a patch and will send out when it is ready. Let me know 
> if you have any questions or feedbacks.
> 
> Thanks
> 
> Steven
> 
> [1] https://opensource.apple.com/source/clang/clang-800.0.38
> [2] http://lists.llvm.org/pipermail/llvm-dev/2016-November/106995.html
> [3] 
>
https://opensource.apple.com/source/clang/clang-800.0.38/src/lib/ObjCMetadata/
> [4] 
>
https://opensource.apple.com/source/clang/clang-800.0.38/src/include/llvm/ObjCMetadata/
> [5] 
>
https://opensource.apple.com/source/clang/clang-800.0.38/src/tools/api-analyzer/
> 
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

Hi David

There is an abstraction layer to allow runtime-specific implementations but it
is possible that you need to tweak some interfaces to make it working for all
your use cases. I design the abstraction to accommodate both macho object file
and llvm bitcode, which are two completely different underlying implementations.

I just post the code review: https://reviews.llvm.org/D59945
<https://reviews.llvm.org/D59945> and I also added you as code reviewer.
Let me know if you have any comments and suggestions.

Steven
> On Mar 26, 2019, at 1:39 AM, David Chisnall via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Hi Steven,
> 
> This sounds like a very useful tool.  In Clang, we've been very careful
to maintain an abstraction layer over different Objective-C runtime
implementations.  Your email talked only about the Apple runtime - do you
already have an abstraction layer for the runtime-specific details, or is this
something that will need to be added prior to upstreaming?
> 
> David
> 
> On 22/03/2019 18:34, Steven Wu via llvm-dev wrote:
>> Overview
>> ======>> We are planning to open source our internal
implementation of Objective-C metadata reader library to LLVM. Unlike the C
style objective-c metadata reader in llvm-objdump, it is designed to be a
reusable C++ library to parse objective-c metadata from binaries with LLVM style
error handling. The current implementation has an interface that resembles
libObject in and it supports parsing objective-c metadata from MachO object file
or LLVM bitcode that is generated by clang/swift compiler. It has the
flexibility to be extended to support other objective-c metadata format if
needed.
>> You can find an old open-source version of implementation here[1]. The
version we planned to open source has the same design but it will come with some
bug fixes and improvements, and it will be rebased to the trunk.
>> Background
>> ========>> We think it is very beneficial to have an objective-c
metadata library in LLVM. We use the library in many tools, including tapi which
we are currently in the process of open sourcing. Some have been asking for this
libraries on the list or off the list to use it in the contexts like JIT[2] or
macports.
>> Originally, the open source of this library is delayed because we
thought it would be good if we can achieve following two goals before
upstreaming:
>>  * An improved libObject interface: we are expecting libObject
>>    interfaces to be changed which ObjCMetadata library should be
>>    updated to match.
>>  * Replace the parser inside llvm-objdump.
>> The first one has been talked for years but never put into action. The
second one requires a strict output match because llvm-objdump is a replacement
for otool-classic on darwin platform, and doing so will require LLVM
infrastructure improvements on how to handle errors. With some discussion, we
believe both tasks can be done in trunk when it is time, rather than waiting for
all the requirements are met.
>> Design
>> ====>> ObjCMetadata library comes with headers (in
llvm/include/llvm/ObjCMetadata)[3], implementations (in
llvm/lib/ObjCMetadata)[4] and we will also implement a tool similar to a
stripped down version of [5] for testing.
>> Here is an example how to use the library by extracting and printing
the objc class names:
>> ```
>>   MachOMetadata ObjCInfo(InputObject);
>> *if* (*auto* ObjCClasses = ObjCInfo.classes()) {
>> *for* (*auto* c : *ObjCClasses) {
>> *auto* ObjCClass = *c;
>> *if* (!ObjCClass) {
>>         handleError(ObjCClass.takeError());
>> *continue*;
>>       }
>> *auto* name = ObjCClass->getName();
>> *if* (!name) {
>>         handleError(name.takeError());
>> *continue*;
>>       }
>>       outs() << *name << "\n";
>>   }
>> ```
>> I am preparing a patch and will send out when it is ready. Let me know
if you have any questions or feedbacks.
>> Thanks
>> Steven
>> [1] https://opensource.apple.com/source/clang/clang-800.0.38
>> [2] http://lists.llvm.org/pipermail/llvm-dev/2016-November/106995.html
>> [3]
https://opensource.apple.com/source/clang/clang-800.0.38/src/lib/ObjCMetadata/
>> [4]
https://opensource.apple.com/source/clang/clang-800.0.38/src/include/llvm/ObjCMetadata/
>> [5]
https://opensource.apple.com/source/clang/clang-800.0.38/src/tools/api-analyzer/
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190328/dca8bff1/attachment.html>