llvm dev - Jul 2019 - Python build dependency in LLVM and/or clang?

Hello llvm-devs,

I'm currently starting to look at implementing compiler intrinsics for the
Arm MVE vector instruction set.

In a similar sort of style to the existing NEON intrinsics, this is going to
involve describing the set of functions needed in Tablegen, and then processing
that data into a header file and some bits and pieces to compile into clang (the
list of builtins, data tables needed by Sema or CodeGen, etc). But the
differences between MVE and NEON are large enough - particularly the predication
system - that I'm going to need new Tablegen backends, rather than reusing
the NEON ones.

I would like, if possible, to write these Tablegen backends in Python, by
consuming the output of the 'llvm-tblgen -dump-json' feature I added to
Tablegen last year. I think they'll be simpler and more concise that way
than in C++, not to mention quicker to develop.

But before I do that, I wanted to check whether there would be any objection on
grounds of build dependencies. Is it acceptable to require Python as part of the
normal build process for LLVM? Or is Python something that can be used in places
like the test suite and developer utilities, but has to be kept out of the main
build to make bootstrapping as easy as possible? I looked for a written policy
on this, but didn't find one.

Cheers,
Simon

On Fri, Jul 05, 2019 at 10:43:12AM +0000, Simon Tatham via llvm-dev
wrote:
> But before I do that, I wanted to check whether there would be any
> objection on grounds of build dependencies. Is it acceptable to require
Python
> as part of the normal build process for LLVM?
At the moment, Python is not truely necessary for building LLVM. The
cmake build system depends on it for some parts, but that's all. I would
be very sad for that to change.

Joerg

> -----Original Message-----
> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of
Joerg
> Sonnenberger via llvm-dev
> Sent: Friday, July 05, 2019 8:01 AM
> To: llvm-dev at lists.llvm.org
> Subject: Re: [llvm-dev] Python build dependency in LLVM and/or clang?
> 
> On Fri, Jul 05, 2019 at 10:43:12AM +0000, Simon Tatham via llvm-dev wrote:
> > But before I do that, I wanted to check whether there would be any
> > objection on grounds of build dependencies. Is it acceptable to
require
> Python
> > as part of the normal build process for LLVM?
> 
> At the moment, Python is not truely necessary for building LLVM. The
> cmake build system depends on it for some parts, but that's all. I
would
> be very sad for that to change.
Technically true, but Python is required to run the 'lit' tests which is
a pretty fundamental operation within LLVM. I can't imagine a situation
where I would want to run a build without running check-whatever after.
So really, you do have to have Python already.
--paulr
> 
> Joerg
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

On Fri, Jul 5, 2019 at 3:43 AM Simon Tatham via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hello llvm-devs,
>
> I'm currently starting to look at implementing compiler intrinsics for
the
> Arm MVE vector instruction set.
>
> In a similar sort of style to the existing NEON intrinsics, this is going
> to involve describing the set of functions needed in Tablegen, and then
> processing that data into a header file and some bits and pieces to compile
> into clang (the list of builtins, data tables needed by Sema or CodeGen,
> etc). But the differences between MVE and NEON are large enough -
> particularly the predication system - that I'm going to need new
Tablegen
> backends, rather than reusing the NEON ones.
>
> I would like, if possible, to write these Tablegen backends in Python, by
> consuming the output of the 'llvm-tblgen -dump-json' feature I
added to
> Tablegen last year. I think they'll be simpler and more concise that
way
> than in C++, not to mention quicker to develop.
>
> But before I do that, I wanted to check whether there would be any
> objection on grounds of build dependencies. Is it acceptable to require
> Python as part of the normal build process for LLVM? Or is Python something
> that can be used in places like the test suite and developer utilities, but
> has to be kept out of the main build to make bootstrapping as easy as
> possible? I looked for a written policy on this, but didn't find one.

More than the bootstrapping question or adding a dependency to the build,
I'm wary of using Python pervasively with respect to maintenance and
readability of the code. While convenient for small scripts, the lack of
static types is making it hard to understand and track the code beyond this
simple use-case. We don't have a style guide for python, the tools
(formatting, indexing, IDEs, ...) used for C++ development in LLVM aren't
providing the same experience with Python, etc.

-- 
Mehdi
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190710/e6e2e28b/attachment.html>

Am Fr., 5. Juli 2019 um 07:01 Uhr schrieb Joerg Sonnenberger via
llvm-dev <llvm-dev at lists.llvm.org>:
>
> On Fri, Jul 05, 2019 at 10:43:12AM +0000, Simon Tatham via llvm-dev wrote:
> > But before I do that, I wanted to check whether there would be any
> > objection on grounds of build dependencies. Is it acceptable to
require Python
> > as part of the normal build process for LLVM?
>
> At the moment, Python is not truely necessary for building LLVM. The
> cmake build system depends on it for some parts, but that's all. I
would
> be very sad for that to change.
llvm-build (LLVMBuild.txt) requires python. The LLVM build will abort with

   message(FATAL_ERROR
    "Unable to find Python interpreter, required for builds and testing.

    Please install Python or specify the PYTHON_EXECUTABLE CMake
variable.")

without it.

Michael

Hi Simon,

Just a reminder: TableGen is a huge build bottleneck on some machines. I worry
that dumping JSON and then pushing that through Python would make this
bottleneck much worse. Consider this: TableGen performance is so critical that
there is an entire build option just to force it to be built with optimizations
when creating debug builds: LLVM_OPTIMIZED_TABLEGEN

Dave
> On Jul 5, 2019, at 12:43 PM, Simon Tatham via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Hello llvm-devs,
> 
> I'm currently starting to look at implementing compiler intrinsics for
the Arm MVE vector instruction set.
> 
> In a similar sort of style to the existing NEON intrinsics, this is going
to involve describing the set of functions needed in Tablegen, and then
processing that data into a header file and some bits and pieces to compile into
clang (the list of builtins, data tables needed by Sema or CodeGen, etc). But
the differences between MVE and NEON are large enough - particularly the
predication system - that I'm going to need new Tablegen backends, rather
than reusing the NEON ones.
> 
> I would like, if possible, to write these Tablegen backends in Python, by
consuming the output of the 'llvm-tblgen -dump-json' feature I added to
Tablegen last year. I think they'll be simpler and more concise that way
than in C++, not to mention quicker to develop.
> 
> But before I do that, I wanted to check whether there would be any
objection on grounds of build dependencies. Is it acceptable to require Python
as part of the normal build process for LLVM? Or is Python something that can be
used in places like the test suite and developer utilities, but has to be kept
out of the main build to make bootstrapping as easy as possible? I looked for a
written policy on this, but didn't find one.
> 
> Cheers,
> Simon
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev