llvm dev - Jul 2015 - [LLVMdev] LLVM parsers for popular languages? - Python, Rust, Go

Thanks, happy to of confirmed.

With that in mind, will use the AST modules provided by the languages (with
the exception of libclang for C++).

Antoine: Am aware of Numba, nice job there BTW. So is there a [decoupled]
LLVM parser which I can use to read Python files and analyse objects
(including computing their attributes in OO and setattr scenarios)?

On Wed, Jul 1, 2015 at 10:23 PM, Antoine Pitrou <antoine at python.org>
wrote:
>
> Hi,
>
> Alec Taylor <alec.taylor6 <at> gmail.com> writes:
> >
> > Would be good to have Python, Rust and Go.Are there any LLVM parsers
> > around for these popular languages?
>
> A programming language is much more than a parser and AST.  It has
> specific semantics, and a runtime (in the case of Python, the runtime is
> very large as it hosts a lot of functionality).
>
> So it wouldn't make much sense to have "just a parser".
>
> However, if you are looking for an implementation of a subset of Python
> using LLVM, you can take a look at Numba: http://numba.pydata.org/
>
> (disclaimer: I am part of the Numba team)
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150704/9d71e72c/attachment.html>

Alec Taylor <alec.taylor6 <at> gmail.com>
writes:
> 
> So is there a [decoupled] LLVM parser which I can use to read Python files
and analyse objects (including computing their attributes in OO and setattr
scenarios)?

There isn't. We simply let Python parse the JITted code itself. The parser
is written in C, and is in the CPython code base. If you want to tinker
with that, there is a doc at https://docs.python.org/devguide/compiler.html

Regards

Antoine.

No worries, I think the ast <https://docs.python.org/2/library/ast.html>
module will suffice for now.

On Sat, Jul 4, 2015 at 6:49 PM, Antoine Pitrou <antoine at python.org>
wrote:
> Alec Taylor <alec.taylor6 <at> gmail.com> writes:
> >
> > So is there a [decoupled] LLVM parser which I can use to read Python
> files
> and analyse objects (including computing their attributes in OO and setattr
> scenarios)?
>
> There isn't. We simply let Python parse the JITted code itself. The
parser
> is written in C, and is in the CPython code base. If you want to tinker
> with that, there is a doc at
> https://docs.python.org/devguide/compiler.html
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150704/87682a26/attachment.html>

There is also the Pyston project from Dropbox. Presumably that includes a
Python parser.

I'm not affiliated with the project.

On Sat, Jul 4, 2015 at 2:35 AM, Alec Taylor <alec.taylor6 at gmail.com>
wrote:
> Thanks, happy to of confirmed.
>
> With that in mind, will use the AST modules provided by the languages
> (with the exception of libclang for C++).
>
> Antoine: Am aware of Numba, nice job there BTW. So is there a [decoupled]
> LLVM parser which I can use to read Python files and analyse objects
> (including computing their attributes in OO and setattr scenarios)?
>
> On Wed, Jul 1, 2015 at 10:23 PM, Antoine Pitrou <antoine at
python.org>
> wrote:
>
>>
>> Hi,
>>
>> Alec Taylor <alec.taylor6 <at> gmail.com
>>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__gmail.com&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Mfk2qtn1LTDThVkh6-oGglNfMADXfJdty4_bhmuhMHA&m=gFwnrq1A6b4bDGDoXVANrEYyGDZzyOClT35YGuILpnw&s=PFeGy9X8Vy60g44Moeq7LIisLzx1skqCoTyOllso94I&e=>>
>> writes:
>> >
>> > Would be good to have Python, Rust and Go.Are there any LLVM
parsers
>> > around for these popular languages?
>>
>> A programming language is much more than a parser and AST.  It has
>> specific semantics, and a runtime (in the case of Python, the runtime
is
>> very large as it hosts a lot of functionality).
>>
>> So it wouldn't make much sense to have "just a parser".
>>
>> However, if you are looking for an implementation of a subset of Python
>> using LLVM, you can take a look at Numba: http://numba.pydata.org/
>>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__numba.pydata.org_&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Mfk2qtn1LTDThVkh6-oGglNfMADXfJdty4_bhmuhMHA&m=gFwnrq1A6b4bDGDoXVANrEYyGDZzyOClT35YGuILpnw&s=NZGvzCXyd0A8Yrs45NE2hq5RLlpBSo6pdWk2ul6lkO4&e=>
>>
>> (disclaimer: I am part of the Numba team)
>>
>> Regards
>>
>> Antoine.
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>
>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150704/0e90ff9c/attachment.html>

Yep we have our own parser <https://github.com/vinzenz/libpypa/> and we
would love to see other people use it.  When we looked around at some other
Python parsers we didn't feel like any of them were easy to extract and use
on their own, so we wrote our own and I think were able to keep ours
well-separated.  There are some things that make parsing Python somewhat
difficult to do in a fully project-agnostic way: any syntax errors usually
get thrown as user-level exceptions, you probably don't want to encode the
full set of unicode character names into your parser to handle u"\N{POUND
SIGN}", and the parser has to support calling back into Python code for
supporting custom encodings requested via "# coding" lines.

I think we've done a decent job factoring those things out (they get
provided by your project via callbacks), but you do have to provide those
features or avoid parsing code that would need them.  If you can get the
job done by working in Python using the ast module, I would recommend that.

On Sat, Jul 4, 2015 at 5:58 AM, David Jones <djones at xtreme-eda.com>
wrote:
> There is also the Pyston project from Dropbox. Presumably that includes a
> Python parser.
>
> I'm not affiliated with the project.
>
> On Sat, Jul 4, 2015 at 2:35 AM, Alec Taylor <alec.taylor6 at
gmail.com>
> wrote:
>
>> Thanks, happy to of confirmed.
>>
>> With that in mind, will use the AST modules provided by the languages
>> (with the exception of libclang for C++).
>>
>> Antoine: Am aware of Numba, nice job there BTW. So is there a
[decoupled]
>> LLVM parser which I can use to read Python files and analyse objects
>> (including computing their attributes in OO and setattr scenarios)?
>>
>> On Wed, Jul 1, 2015 at 10:23 PM, Antoine Pitrou <antoine at
python.org>
>> wrote:
>>
>>>
>>> Hi,
>>>
>>> Alec Taylor <alec.taylor6 <at> gmail.com
>>>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__gmail.com&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Mfk2qtn1LTDThVkh6-oGglNfMADXfJdty4_bhmuhMHA&m=gFwnrq1A6b4bDGDoXVANrEYyGDZzyOClT35YGuILpnw&s=PFeGy9X8Vy60g44Moeq7LIisLzx1skqCoTyOllso94I&e=>>
>>> writes:
>>> >
>>> > Would be good to have Python, Rust and Go.Are there any LLVM
parsers
>>> > around for these popular languages?
>>>
>>> A programming language is much more than a parser and AST.  It has
>>> specific semantics, and a runtime (in the case of Python, the
runtime is
>>> very large as it hosts a lot of functionality).
>>>
>>> So it wouldn't make much sense to have "just a
parser".
>>>
>>> However, if you are looking for an implementation of a subset of
Python
>>> using LLVM, you can take a look at Numba: http://numba.pydata.org/
>>>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__numba.pydata.org_&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Mfk2qtn1LTDThVkh6-oGglNfMADXfJdty4_bhmuhMHA&m=gFwnrq1A6b4bDGDoXVANrEYyGDZzyOClT35YGuILpnw&s=NZGvzCXyd0A8Yrs45NE2hq5RLlpBSo6pdWk2ul6lkO4&e=>
>>>
>>> (disclaimer: I am part of the Numba team)
>>>
>>> Regards
>>>
>>> Antoine.
>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>>
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>
>>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150704/234e975d/attachment.html>