llvm dev - Oct 2017 - Adding a third-party dependency in clang-tools-extra

clangd communicates with an editor via JSON-RPC. It parses JSON with
YAMLParser, which is awkward, and generates JSON with printf and friends,
which is miserable. Much of LLVM does things this way, but clangd does it a
lot.

I'd like to try replacing this with a JSON library. nlohmann/json[1] seems
like a reasonable fit: C++11 with exceptions optional, simple build, MIT
license.

I'd propose vendoring it under tools/clang/tools/extra/clangd/nlohmann-json
so there's no question of it "leaking" into runtimes as described
in this
thread[2].
This also means it wouldn't solve llvm's general JSON-parser problem :-)

Any LLVM-level objections or concerns? (Whether that library is the right
technical choice for clangd can be discussed elsewhere, I think)
If anyone wants to argue that we *shouldn't* bury it in
clang/tools/extra/clangd, that's fine too!

Cheers, Sam

[1] https://github.com/nlohmann/json/blob/develop/src/json.hpp
[2] https://groups.google.com/d/msg/llvm-dev/2JHX3smXTpE/U-E32Yg0AAAJ
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171019/a09b572b/attachment.html>

On Thu, Oct 19, 2017 at 3:48 AM Sam McCall <sammccall at google.com>
wrote:
> clangd communicates with an editor via JSON-RPC. It parses JSON with
> YAMLParser, which is awkward, and generates JSON with printf and friends,
> which is miserable. Much of LLVM does things this way, but clangd does it a
> lot.
>
> I'd like to try replacing this with a JSON library. nlohmann/json[1]
seems
> like a reasonable fit: C++11 with exceptions optional, simple build, MIT
> license.
>
> I'd propose vendoring it under
> tools/clang/tools/extra/clangd/nlohmann-json so there's no question of
it
> "leaking" into runtimes as described in this thread[2].
> This also means it wouldn't solve llvm's general JSON-parser
problem :-)
>
> Any LLVM-level objections or concerns? (Whether that library is the right
> technical choice for clangd can be discussed elsewhere, I think)
> If anyone wants to argue that we *shouldn't* bury it in
> clang/tools/extra/clangd, that's fine too!
>
Generally, I feel like we have continued to find JSON and YAML uses in
LLVM. I think it would be a shame to have more code in that space in the
repository.

But that brings us to another problem: outside of tests, we really
shouldn't add more third-party code with different licenses to LLVM. As a
compiler, LLVM has rather unique licensing requirements. So while this
might be "fine" inside of clangd, if it moves elsewhere (and in many
ways
it should!) it would become a problem. Worse, it might be easily missed, or
accidentally end up being used elsewhere.

I would personally be fairly reluctant to take this on unless there is a
pretty huge reason why it is needed. For example, if we had a absolute need
to do proper XML parsing and manipulation, the amount of code required for
that would be untenable without using one of the existing XML libraries.
LLDB for example actually does use an XML library IIRC.

I'm hoping that the problem domain here is substantially simpler and it is
tenable (if never really appealing) to just roll our own....
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171019/8e84a2de/attachment.html>

On Thu, Oct 19, 2017 at 9:50 AM Chandler Carruth <chandlerc at google.com>
wrote:
> On Thu, Oct 19, 2017 at 3:48 AM Sam McCall <sammccall at google.com>
wrote:
>
>> clangd communicates with an editor via JSON-RPC. It parses JSON with
>> YAMLParser, which is awkward, and generates JSON with printf and
friends,
>> which is miserable. Much of LLVM does things this way, but clangd does
it a
>> lot.
>>
>> I'd like to try replacing this with a JSON library.
nlohmann/json[1]
>> seems like a reasonable fit: C++11 with exceptions optional, simple
build,
>> MIT license.
>>
>> I'd propose vendoring it under
>> tools/clang/tools/extra/clangd/nlohmann-json so there's no question
of it
>> "leaking" into runtimes as described in this thread[2].
>> This also means it wouldn't solve llvm's general JSON-parser
problem :-)
>>
>> Any LLVM-level objections or concerns? (Whether that library is the
right
>> technical choice for clangd can be discussed elsewhere, I think)
>> If anyone wants to argue that we *shouldn't* bury it in
>> clang/tools/extra/clangd, that's fine too!
>>
>
> Generally, I feel like we have continued to find JSON and YAML uses in
> LLVM. I think it would be a shame to have more code in that space in the
> repository.
>
> But that brings us to another problem: outside of tests, we really
> shouldn't add more third-party code with different licenses to LLVM. As
a
> compiler, LLVM has rather unique licensing requirements. So while this
> might be "fine" inside of clangd, if it moves elsewhere (and in
many ways
> it should!) it would become a problem. Worse, it might be easily missed, or
> accidentally end up being used elsewhere.
>
> I would personally be fairly reluctant to take this on unless there is a
> pretty huge reason why it is needed. For example, if we had a absolute need
> to do proper XML parsing and manipulation, the amount of code required for
> that would be untenable without using one of the existing XML libraries.
> LLDB for example actually does use an XML library IIRC.
>
> I'm hoping that the problem domain here is substantially simpler and it
is
> tenable (if never really appealing) to just roll our own....
>
We already had a JSON parser at some point, but that was deleted as YAML
was considered a replacement (https://reviews.llvm.org/rL146735).
Regarding writing out JSON with a library, I'd be curious in the design - I
do agree with Chandler that all designs I'd come up with are rather simple.

That said, I'd be curious to see how code that is written against
nlohmann/json
would look compared to code that's currently written.

If compelling, could an alternative be to make it a build time dep for
folks wanting to build clangd, as opposed to putting it in svn?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171019/ffd49dd0/attachment.html>

On Oct 19, 2017 6:50 PM, "Chandler Carruth" <chandlerc at
google.com> wrote:

On Thu, Oct 19, 2017 at 3:48 AM Sam McCall <sammccall at google.com>
wrote:
> clangd communicates with an editor via JSON-RPC. It parses JSON with
> YAMLParser, which is awkward, and generates JSON with printf and friends,
> which is miserable. Much of LLVM does things this way, but clangd does it a
> lot.
>
> I'd like to try replacing this with a JSON library. nlohmann/json[1]
seems
> like a reasonable fit: C++11 with exceptions optional, simple build, MIT
> license.
>
> I'd propose vendoring it under
tools/clang/tools/extra/clangd/nlohmann-json
> so there's no question of it "leaking" into runtimes as
described in this
> thread[2].
> This also means it wouldn't solve llvm's general JSON-parser
problem :-)
>
> Any LLVM-level objections or concerns? (Whether that library is the right
> technical choice for clangd can be discussed elsewhere, I think)
> If anyone wants to argue that we *shouldn't* bury it in
> clang/tools/extra/clangd, that's fine too!
>
Generally, I feel like we have continued to find JSON and YAML uses in
LLVM. I think it would be a shame to have more code in that space in the
repository.

But that brings us to another problem: outside of tests, we really
shouldn't add more third-party code with different licenses to LLVM. As a
compiler, LLVM has rather unique licensing requirements. So while this
might be "fine" inside of clangd, if it moves elsewhere (and in many
ways
it should!) it would become a problem. Worse, it might be easily missed, or
accidentally end up being used elsewhere.

Fair enough - we do need JSON facilities in many places.
I'm both surprised and unsurprised that licensing is a concern here :-)

I would personally be fairly reluctant to take this on unless there is a
pretty huge reason why it is needed. For example, if we had a absolute need
to do proper XML parsing and manipulation, the amount of code required for
that would be untenable without using one of the existing XML libraries.
LLDB for example actually does use an XML library IIRC.

I'm hoping that the problem domain here is substantially simpler and it is
tenable (if never really appealing) to just roll our own....
> Of course, my first inclination was to start writing one, and I had to
restrain myself!

Happy to have a crack at this and start a bikeshed thread over the design.
My main concerns:
 - compromising ease-of-use to satisfy every use case in LLVM. In
particular, I really want an eager parser rather than the streaming style
of YAMLparser.
 - a new library will need a test suite, benchmarks etc - sounds like using
third-party code for this is OK though.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171019/66642ad2/attachment.html>