llvm dev - Sep 2021 - [cfe-dev] [RFC] clang support for API information generation in JSON

> On Sep 1, 2021, at 11:26 AM, Tom Stellard <tstellar at redhat.com>
wrote:
> 
> On 9/1/21 11:18 AM, Zixu Wang via cfe-dev wrote:
>> Hi All!
>> I’m writing to propose clang-extract-api, a tool to collect and
serialize API information from header files, for example function signatures,
Objective-C interfaces and protocols, and inline documentation comments. We hope
it could help future tools to understand clang-based language APIs without
needing to dig into the AST themselves.
> 
> Would this tool be able to provide the same functionality as tools
> like abi-compliance-checker[1] and libabigail[2], that extra ABI/API
> information from debuginfo?
> 
> -Tom
> 
> [1] https://github.com/lvc/abi-compliance-checker
> [2] https://sourceware.org/libabigail/
Hi Tom!

I’m not really familiar with those tools but a brief look seems to suggest that
they operate on compiled binaries. We propose clang-extract-api to directly work
on and extract information from the parsed AST of the header source files so I’m
not quite sure how to compare these tools. Debuginfo lives in the compiled
binary so clang-extract-api doesn’t look into it. However, anything about the
API described in the header file could potentially be extracted.

Do you have a specific example of information you want to get from
abi-compliance-checker and libabigail?

Zixu
> 
>> *Background
>> *
>> *Motivation*
>> Library and SDK providers may find it useful to be able to create and
inspect a “snapshot” of APIs they expose, for example, to check for
API/ABI-breaking changes between two versions, or to automate generating
documentation for the APIs. Here is a list of examples of information we want to
extract from APIs:
>> • the name (spelling/mangled) of the symbol;
>> • the unique identifier of the symbol, for example the Unified Symbol
Resolution (USR);
>> • the source location of the API declaration (file, line, column);
>> • access control of the API (public/private/protected);
>> • availability (available/unavailable/deprecated);
>> • function signatures (return/parameters);
>> • documentation comments attached to a symbol;
>> • relations with other symbols (class methods, typedef relations,
struct data fields, enum constants, etc.)
>> Since these API information is available in the header files, which
declare and distribute the APIs, we can implement a tool to extract them without
invoking a compilation of the whole project to enable easy access to the
information for tooling.
>> *Existing solutions*
>> While there are some existing solutions in clang to dump symbols or AST
information, they either expose unnecessary low-level details or fail to provide
enough information of APIs. For example, clang -ast-dump dumps low-level details
for all declarations for debug purposes and the output is not machine-parsable.
Doxygen also extracts documentation comments and other information from API
declarations, but its output is rendered documentation in web formats which is
not flexible for other uses and tools.
>> *Proposal*
>> We propose to implement this tool as a new frontend action invoked by
`clang -extract-api` as show in the example below.
>> clang -extract-api \
>>   header.h [more_header.h ...] or a filelist\
>>   -isysroot <SDK> \
>>   -target <TARGET> \
>>   -I <INCLUDE PATH> \
>>   -isystem <SYS INCLUDE PATH> \
>>   ...
>>   -o output.json
>> It takes in the header file(s) or a filelist file containing paths to
the header file(s) as the input. The header files will be parsed by clang and
the extract-api action will visit the AST to extract needed information and
serialize to a JSON output. Please find an example input and output attached.
>> The example output is based on the symbol graph format that's
already used by Swift for serializing symbol information and their relations.
This format can represent the required API information and is flexible and
extendable as demonstrated in the example so we think it's a good starting
point.
>> We are excited about this idea and its potential uses, and we’d love to
hear feedback and suggestions!
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>

On 9/1/21 11:40 AM, Zixu Wang wrote:
> 
> 
>> On Sep 1, 2021, at 11:26 AM, Tom Stellard <tstellar at
redhat.com> wrote:
>>
>> On 9/1/21 11:18 AM, Zixu Wang via cfe-dev wrote:
>>> Hi All!
>>> I’m writing to propose clang-extract-api, a tool to collect and
serialize API information from header files, for example function signatures,
Objective-C interfaces and protocols, and inline documentation comments. We hope
it could help future tools to understand clang-based language APIs without
needing to dig into the AST themselves.
>>
>> Would this tool be able to provide the same functionality as tools
>> like abi-compliance-checker[1] and libabigail[2], that extra ABI/API
>> information from debuginfo?
>>
>> -Tom
>>
>> [1] https://github.com/lvc/abi-compliance-checker
>> [2] https://sourceware.org/libabigail/
> 
> Hi Tom!
> 
> I’m not really familiar with those tools but a brief look seems to suggest
that they operate on compiled binaries. We propose clang-extract-api to directly
work on and extract information from the parsed AST of the header source files
so I’m not quite sure how to compare these tools. Debuginfo lives in the
compiled binary so clang-extract-api doesn’t look into it. However, anything
about the API described in the header file could potentially be extracted.
> 
> Do you have a specific example of information you want to get from
abi-compliance-checker and libabigail?
> 
We have a CI job[1] in the release branch that runs the abi-compliance-checker
tool to
ensure that we don't accidentally change the ABI/API of libclang.so.  Is
this something
your tool could be used for ?

-Tom

[1]
https://github.com/llvm/llvm-project/blob/release/13.x/.github/workflows/libclang-abi-tests.yml
> Zixu
> 
>>
>>> *Background
>>> *
>>> *Motivation*
>>> Library and SDK providers may find it useful to be able to create
and inspect a “snapshot” of APIs they expose, for example, to check for
API/ABI-breaking changes between two versions, or to automate generating
documentation for the APIs. Here is a list of examples of information we want to
extract from APIs:
>>> • the name (spelling/mangled) of the symbol;
>>> • the unique identifier of the symbol, for example the Unified
Symbol Resolution (USR);
>>> • the source location of the API declaration (file, line, column);
>>> • access control of the API (public/private/protected);
>>> • availability (available/unavailable/deprecated);
>>> • function signatures (return/parameters);
>>> • documentation comments attached to a symbol;
>>> • relations with other symbols (class methods, typedef relations,
struct data fields, enum constants, etc.)
>>> Since these API information is available in the header files, which
declare and distribute the APIs, we can implement a tool to extract them without
invoking a compilation of the whole project to enable easy access to the
information for tooling.
>>> *Existing solutions*
>>> While there are some existing solutions in clang to dump symbols or
AST information, they either expose unnecessary low-level details or fail to
provide enough information of APIs. For example, clang -ast-dump dumps low-level
details for all declarations for debug purposes and the output is not
machine-parsable. Doxygen also extracts documentation comments and other
information from API declarations, but its output is rendered documentation in
web formats which is not flexible for other uses and tools.
>>> *Proposal*
>>> We propose to implement this tool as a new frontend action invoked
by `clang -extract-api` as show in the example below.
>>> clang -extract-api \
>>>    header.h [more_header.h ...] or a filelist\
>>>    -isysroot <SDK> \
>>>    -target <TARGET> \
>>>    -I <INCLUDE PATH> \
>>>    -isystem <SYS INCLUDE PATH> \
>>>    ...
>>>    -o output.json
>>> It takes in the header file(s) or a filelist file containing paths
to the header file(s) as the input. The header files will be parsed by clang and
the extract-api action will visit the AST to extract needed information and
serialize to a JSON output. Please find an example input and output attached.
>>> The example output is based on the symbol graph format that's
already used by Swift for serializing symbol information and their relations.
This format can represent the required API information and is flexible and
extendable as demonstrated in the example so we think it's a good starting
point.
>>> We are excited about this idea and its potential uses, and we’d
love to hear feedback and suggestions!
>>> _______________________________________________
>>> cfe-dev mailing list
>>> cfe-dev at lists.llvm.org
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>
>