Kevin Modzelewski
2015-Jul-04 23:58 UTC
[LLVMdev] LLVM parsers for popular languages? - Python, Rust, Go
Yep we have our own parser <https://github.com/vinzenz/libpypa/> and we
would love to see other people use it. When we looked around at some other
Python parsers we didn't feel like any of them were easy to extract and use
on their own, so we wrote our own and I think were able to keep ours
well-separated. There are some things that make parsing Python somewhat
difficult to do in a fully project-agnostic way: any syntax errors usually
get thrown as user-level exceptions, you probably don't want to encode the
full set of unicode character names into your parser to handle u"\N{POUND
SIGN}", and the parser has to support calling back into Python code for
supporting custom encodings requested via "# coding" lines.
I think we've done a decent job factoring those things out (they get
provided by your project via callbacks), but you do have to provide those
features or avoid parsing code that would need them. If you can get the
job done by working in Python using the ast module, I would recommend that.
On Sat, Jul 4, 2015 at 5:58 AM, David Jones <djones at xtreme-eda.com>
wrote:
> There is also the Pyston project from Dropbox. Presumably that includes a
> Python parser.
>
> I'm not affiliated with the project.
>
> On Sat, Jul 4, 2015 at 2:35 AM, Alec Taylor <alec.taylor6 at
gmail.com>
> wrote:
>
>> Thanks, happy to of confirmed.
>>
>> With that in mind, will use the AST modules provided by the languages
>> (with the exception of libclang for C++).
>>
>> Antoine: Am aware of Numba, nice job there BTW. So is there a
[decoupled]
>> LLVM parser which I can use to read Python files and analyse objects
>> (including computing their attributes in OO and setattr scenarios)?
>>
>> On Wed, Jul 1, 2015 at 10:23 PM, Antoine Pitrou <antoine at
python.org>
>> wrote:
>>
>>>
>>> Hi,
>>>
>>> Alec Taylor <alec.taylor6 <at> gmail.com
>>>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__gmail.com&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Mfk2qtn1LTDThVkh6-oGglNfMADXfJdty4_bhmuhMHA&m=gFwnrq1A6b4bDGDoXVANrEYyGDZzyOClT35YGuILpnw&s=PFeGy9X8Vy60g44Moeq7LIisLzx1skqCoTyOllso94I&e=>>
>>> writes:
>>> >
>>> > Would be good to have Python, Rust and Go.Are there any LLVM
parsers
>>> > around for these popular languages?
>>>
>>> A programming language is much more than a parser and AST. It has
>>> specific semantics, and a runtime (in the case of Python, the
runtime is
>>> very large as it hosts a lot of functionality).
>>>
>>> So it wouldn't make much sense to have "just a
parser".
>>>
>>> However, if you are looking for an implementation of a subset of
Python
>>> using LLVM, you can take a look at Numba: http://numba.pydata.org/
>>>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__numba.pydata.org_&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Mfk2qtn1LTDThVkh6-oGglNfMADXfJdty4_bhmuhMHA&m=gFwnrq1A6b4bDGDoXVANrEYyGDZzyOClT35YGuILpnw&s=NZGvzCXyd0A8Yrs45NE2hq5RLlpBSo6pdWk2ul6lkO4&e=>
>>>
>>> (disclaimer: I am part of the Numba team)
>>>
>>> Regards
>>>
>>> Antoine.
>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>>
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>
>>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150704/234e975d/attachment.html>
Alec Taylor
2015-Jul-05 06:08 UTC
[LLVMdev] LLVM parsers for popular languages? - Python, Rust, Go
Thanks, that looks like an interesting project. How do I build it? I've tried: $ cd libpypa && mkdir build && cd $_ && cmake .. -G 'Unix Makefiles' && make But that didn't give me the `parser-test` binary for experimenting with (as per your README usage). On Sun, Jul 5, 2015 at 9:58 AM, Kevin Modzelewski <kevmod at gmail.com> wrote:> Yep we have our own parser <https://github.com/vinzenz/libpypa/> and we > would love to see other people use it. When we looked around at some other > Python parsers we didn't feel like any of them were easy to extract and use > on their own, so we wrote our own and I think were able to keep ours > well-separated. There are some things that make parsing Python somewhat > difficult to do in a fully project-agnostic way: any syntax errors usually > get thrown as user-level exceptions, you probably don't want to encode the > full set of unicode character names into your parser to handle u"\N{POUND > SIGN}", and the parser has to support calling back into Python code for > supporting custom encodings requested via "# coding" lines. > > I think we've done a decent job factoring those things out (they get > provided by your project via callbacks), but you do have to provide those > features or avoid parsing code that would need them. If you can get the > job done by working in Python using the ast module, I would recommend that. > > On Sat, Jul 4, 2015 at 5:58 AM, David Jones <djones at xtreme-eda.com> wrote: > >> There is also the Pyston project from Dropbox. Presumably that includes a >> Python parser. >> >> I'm not affiliated with the project. >> >> On Sat, Jul 4, 2015 at 2:35 AM, Alec Taylor <alec.taylor6 at gmail.com> >> wrote: >> >>> Thanks, happy to of confirmed. >>> >>> With that in mind, will use the AST modules provided by the languages >>> (with the exception of libclang for C++). >>> >>> Antoine: Am aware of Numba, nice job there BTW. So is there a >>> [decoupled] LLVM parser which I can use to read Python files and analyse >>> objects (including computing their attributes in OO and setattr scenarios)? >>> >>> On Wed, Jul 1, 2015 at 10:23 PM, Antoine Pitrou <antoine at python.org> >>> wrote: >>> >>>> >>>> Hi, >>>> >>>> Alec Taylor <alec.taylor6 <at> gmail.com >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__gmail.com&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Mfk2qtn1LTDThVkh6-oGglNfMADXfJdty4_bhmuhMHA&m=gFwnrq1A6b4bDGDoXVANrEYyGDZzyOClT35YGuILpnw&s=PFeGy9X8Vy60g44Moeq7LIisLzx1skqCoTyOllso94I&e=>> >>>> writes: >>>> > >>>> > Would be good to have Python, Rust and Go.Are there any LLVM parsers >>>> > around for these popular languages? >>>> >>>> A programming language is much more than a parser and AST. It has >>>> specific semantics, and a runtime (in the case of Python, the runtime is >>>> very large as it hosts a lot of functionality). >>>> >>>> So it wouldn't make much sense to have "just a parser". >>>> >>>> However, if you are looking for an implementation of a subset of Python >>>> using LLVM, you can take a look at Numba: http://numba.pydata.org/ >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__numba.pydata.org_&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Mfk2qtn1LTDThVkh6-oGglNfMADXfJdty4_bhmuhMHA&m=gFwnrq1A6b4bDGDoXVANrEYyGDZzyOClT35YGuILpnw&s=NZGvzCXyd0A8Yrs45NE2hq5RLlpBSo6pdWk2ul6lkO4&e=> >>>> >>>> (disclaimer: I am part of the Numba team) >>>> >>>> Regards >>>> >>>> Antoine. >>>> >>>> >>>> _______________________________________________ >>>> LLVM Developers mailing list >>>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >>>> >>> >>> >>> _______________________________________________ >>> LLVM Developers mailing list >>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >>> >>> >> >> _______________________________________________ >> LLVM Developers mailing list >> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150705/6a8835c9/attachment.html>
Kevin Modzelewski
2015-Jul-06 02:10 UTC
[LLVMdev] LLVM parsers for popular languages? - Python, Rust, Go
Hmm I'm not sure; might be best to have the discussion at https://gitter.im/vinzenz/libpypa where you can reach the parser's author. On Sat, Jul 4, 2015 at 11:08 PM, Alec Taylor <alec.taylor6 at gmail.com> wrote:> Thanks, that looks like an interesting project. How do I build it? > > I've tried: > $ cd libpypa && mkdir build && cd $_ && cmake .. -G 'Unix Makefiles' && > make > > But that didn't give me the `parser-test` binary for experimenting with > (as per your README usage). > > On Sun, Jul 5, 2015 at 9:58 AM, Kevin Modzelewski <kevmod at gmail.com> > wrote: > >> Yep we have our own parser <https://github.com/vinzenz/libpypa/> and we >> would love to see other people use it. When we looked around at some other >> Python parsers we didn't feel like any of them were easy to extract and use >> on their own, so we wrote our own and I think were able to keep ours >> well-separated. There are some things that make parsing Python somewhat >> difficult to do in a fully project-agnostic way: any syntax errors usually >> get thrown as user-level exceptions, you probably don't want to encode the >> full set of unicode character names into your parser to handle u"\N{POUND >> SIGN}", and the parser has to support calling back into Python code for >> supporting custom encodings requested via "# coding" lines. >> >> I think we've done a decent job factoring those things out (they get >> provided by your project via callbacks), but you do have to provide those >> features or avoid parsing code that would need them. If you can get the >> job done by working in Python using the ast module, I would recommend that. >> >> On Sat, Jul 4, 2015 at 5:58 AM, David Jones <djones at xtreme-eda.com> >> wrote: >> >>> There is also the Pyston project from Dropbox. Presumably that includes >>> a Python parser. >>> >>> I'm not affiliated with the project. >>> >>> On Sat, Jul 4, 2015 at 2:35 AM, Alec Taylor <alec.taylor6 at gmail.com> >>> wrote: >>> >>>> Thanks, happy to of confirmed. >>>> >>>> With that in mind, will use the AST modules provided by the languages >>>> (with the exception of libclang for C++). >>>> >>>> Antoine: Am aware of Numba, nice job there BTW. So is there a >>>> [decoupled] LLVM parser which I can use to read Python files and analyse >>>> objects (including computing their attributes in OO and setattr scenarios)? >>>> >>>> On Wed, Jul 1, 2015 at 10:23 PM, Antoine Pitrou <antoine at python.org> >>>> wrote: >>>> >>>>> >>>>> Hi, >>>>> >>>>> Alec Taylor <alec.taylor6 <at> gmail.com >>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__gmail.com&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Mfk2qtn1LTDThVkh6-oGglNfMADXfJdty4_bhmuhMHA&m=gFwnrq1A6b4bDGDoXVANrEYyGDZzyOClT35YGuILpnw&s=PFeGy9X8Vy60g44Moeq7LIisLzx1skqCoTyOllso94I&e=>> >>>>> writes: >>>>> > >>>>> > Would be good to have Python, Rust and Go.Are there any LLVM parsers >>>>> > around for these popular languages? >>>>> >>>>> A programming language is much more than a parser and AST. It has >>>>> specific semantics, and a runtime (in the case of Python, the runtime >>>>> is >>>>> very large as it hosts a lot of functionality). >>>>> >>>>> So it wouldn't make much sense to have "just a parser". >>>>> >>>>> However, if you are looking for an implementation of a subset of Python >>>>> using LLVM, you can take a look at Numba: http://numba.pydata.org/ >>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__numba.pydata.org_&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Mfk2qtn1LTDThVkh6-oGglNfMADXfJdty4_bhmuhMHA&m=gFwnrq1A6b4bDGDoXVANrEYyGDZzyOClT35YGuILpnw&s=NZGvzCXyd0A8Yrs45NE2hq5RLlpBSo6pdWk2ul6lkO4&e=> >>>>> >>>>> (disclaimer: I am part of the Numba team) >>>>> >>>>> Regards >>>>> >>>>> Antoine. >>>>> >>>>> >>>>> _______________________________________________ >>>>> LLVM Developers mailing list >>>>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >>>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >>>>> >>>> >>>> >>>> _______________________________________________ >>>> LLVM Developers mailing list >>>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >>>> >>>> >>> >>> _______________________________________________ >>> LLVM Developers mailing list >>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >>> >>> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150705/3d932e0b/attachment.html>