llvm dev - Jan 2020 - [RFC] Python 2 / Python 3 status

Hi folks,

Python2 has reached end of support[0], and many core Python packages are
dropping pthon 2 support[1].

This is a subject that raises periodically on this mlist, with a rather strong
no in 2018 [-1] and a slow move in 2019 [-2, -3].

Even if Python is not a core build requirement, it's used during some
configurations steps (e.g. in the cmake export_executable_symbols function),
for testing, and it is definitevely part of the user experience - several clang
tools depend on it, as well as many lldb features. So it's not only about
utility
scripts.

Fedora has moved to Python3 [2], RHEL8 ships Python3 by default [3] (though
RHEL7 still ships Python2, but it has reached « End of Full Support »).
Latest OSX version also explicitly obsoletes Python2 [4]. Debian always ships
both interpreter - even in stable [5]

There has been an effort in LLVM/lldb/clang/... to make all Python scripts
portable across Python2 and Python3 version [6], but if the documentation and
``GettingStarted.rst`` is to be trust, we only require >=2.7. This may be
considered as a maintenance burden, see for instance this discussion [7]. Plus
LLVM doesn't have a strong conservative culture, e.g. wrt CMake
requirements.

My personal take on this would be to start moving forward. Still supporting both
version this year, but obsoleting Python 2.7 and requiring, say Python 3.6,
starting January 2021 looks like a good compromise.

Any thoughts?

[-3] https://lists.llvm.org/pipermail/llvm-dev/2019-May/132507.html
[-2] http://lists.llvm.org/pipermail/llvm-dev/2019-December/137634.html
[-1] http://lists.llvm.org/pipermail/llvm-dev/2018-January/120826.html]
[0] https://www.python.org/dev/peps/pep-0373/#update
[1] https://python3statement.org/
[2] https://fedoraproject.org/wiki/FinalizingFedoraSwitchtoPython3
[3]
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/8.0_release_notes/index#web-servers-databases-dynamic-languages
[4]
https://developer.apple.com/documentation/macos_release_notes/macos_catalina_10_15_release_notes#3318257)
[5] https://wiki.debian.org/Python
[6]
https://github.com/llvm/llvm-project/search?q=python+compatibility&unscoped_q=python+compatibility&type=Commits
[7] https://reviews.llvm.org/D73011

On Wed, Jan 29, 2020 at 5:26 AM Serge Guelton via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> My personal take on this would be to start moving forward. Still supporting
both
> version this year, but obsoleting Python 2.7 and requiring, say Python 3.6,
> starting January 2021 looks like a good compromise.
I'm relatively new to LLVM so my +1 doesn't count for much, but I am
very supportive of this idea! I maintain a downstream fork of LLVM at
Facebook, and Python 2 will no longer be made available in our
developer environment at some point in 2020. So if LLVM is still
maintaining Python 2-compatible programs in 2021, I'll need to install
a Python 2 interpreter on a personal computer of mine to test any
changes I make, which would be a big inconvenience.
> Even if Python is not a core build requirement, it's used during some
> configurations steps (e.g. in the cmake export_executable_symbols
function),
> for testing, and it is definitevely part of the user experience - several
clang
> tools depend on it, as well as many lldb features. So it's not only
about utility
> scripts.
My understanding is that the build relies upon 232 LLVMBuild.txt
files, which are processed by a Python program llvm-build. I think
that if one uses CMake to build, then Python is in fact a core build
requirement (I'm not sure if gn uses llvm-build?).

I don't want to derail the discussion here, but I have been meaning to
ask: I recall that it was once used more widely, but these days, why
does llvm-build exist? (Again, I'm a beginner in the project, so
forgive my ignorance here!)

I read through some of the code and from what I understand, one of its
main functions is to define the library dependencies of LLVM libraries
like LLVMX86CodeGen or LLVMObject, both as global properties in CMake
(build/LLVMBuild.cmake), and as a C struct that can be used by
llvm-config.cpp (build/tools/llvm-config/LibraryDependencies.inc). But
is there a reason these dependencies couldn't/shouldn't be hardcoded,
instead of being generated by the llvm-build Python executable?

I'm wondering if there's some dynamic build configuration feature of
llvm-build that I'm missing. What makes it hard to remove?
> Any thoughts?
Pardon my tangential questions about llvm-build -- thank you for
proposing this timeline!

- Brian Gesiak

On Wed, Jan 29, 2020 at 2:27 AM Serge Guelton via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> My personal take on this would be to start moving forward. Still
> supporting both
> version this year, but obsoleting Python 2.7 and requiring, say Python 3.6,
> starting January 2021 looks like a good compromise.
>
Sounds good to me. Keeping the window of time during which we support both
Python 2 and 3 as small as possible would be nice.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200129/a72bb7a0/attachment.html>

I don’t know how Python-infested the build system is, but certainly the basic
test suite (lit) is all Python.
Just to add to the notes about distros, on Windows it looks like Visual Studio
2017 comes with Python 3.6.
--paulr

From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of Reid
Kleckner via llvm-dev
Sent: Wednesday, January 29, 2020 1:49 PM
To: Serge Guelton <sguelton at redhat.com>
Cc: llvm-dev <llvm-dev at lists.llvm.org>
Subject: Re: [llvm-dev] [RFC] Python 2 / Python 3 status

On Wed, Jan 29, 2020 at 2:27 AM Serge Guelton via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
My personal take on this would be to start moving forward. Still supporting both
version this year, but obsoleting Python 2.7 and requiring, say Python 3.6,
starting January 2021 looks like a good compromise.

Sounds good to me. Keeping the window of time during which we support both
Python 2 and 3 as small as possible would be nice.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200129/e3ede1c5/attachment.html>

> On Jan 29, 2020, at 5:23 AM, Brian Gesiak via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> On Wed, Jan 29, 2020 at 5:26 AM Serge Guelton via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>> My personal take on this would be to start moving forward. Still
supporting both
>> version this year, but obsoleting Python 2.7 and requiring, say Python
3.6,
>> starting January 2021 looks like a good compromise.
> 
> I'm relatively new to LLVM so my +1 doesn't count for much, but I
am
> very supportive of this idea! I maintain a downstream fork of LLVM at
> Facebook, and Python 2 will no longer be made available in our
> developer environment at some point in 2020. So if LLVM is still
> maintaining Python 2-compatible programs in 2021, I'll need to install
> a Python 2 interpreter on a personal computer of mine to test any
> changes I make, which would be a big inconvenience.
> 
>> Even if Python is not a core build requirement, it's used during
some
>> configurations steps (e.g. in the cmake export_executable_symbols
function),
>> for testing, and it is definitevely part of the user experience -
several clang
>> tools depend on it, as well as many lldb features. So it's not only
about utility
>> scripts.
> 
> My understanding is that the build relies upon 232 LLVMBuild.txt
> files, which are processed by a Python program llvm-build. I think
> that if one uses CMake to build, then Python is in fact a core build
> requirement (I'm not sure if gn uses llvm-build?).
> 
> I don't want to derail the discussion here, but I have been meaning to
> ask: I recall that it was once used more widely, but these days, why
> does llvm-build exist? (Again, I'm a beginner in the project, so
> forgive my ignorance here!)
llvm-build exists for two reasons, one legacy, one still an issue. Back when we
supported both autoconf and CMake llvm-build provided a way to express the build
dependency graph in one place and have both build systems respect it.
Additionally it generates the tables that drive llvm-config. Deprecating
llvm-config in favor of pkg-config and CMake packages has been discussed, but
there has been no decision.

Removing llvm-build is technically feasible even if we continue supporting
llvm-config, but a non-insignificant amount of work to get CMake to generate the
headers needed in the llvm-config build.

-Chris
> I read through some of the code and from what I understand, one of its
> main functions is to define the library dependencies of LLVM libraries
> like LLVMX86CodeGen or LLVMObject, both as global properties in CMake
> (build/LLVMBuild.cmake), and as a C struct that can be used by
> llvm-config.cpp (build/tools/llvm-config/LibraryDependencies.inc). But
> is there a reason these dependencies couldn't/shouldn't be
hardcoded,
> instead of being generated by the llvm-build Python executable?
> 
> I'm wondering if there's some dynamic build configuration feature
of
> llvm-build that I'm missing. What makes it hard to remove?
> 
>> Any thoughts?
> 
> Pardon my tangential questions about llvm-build -- thank you for
> proposing this timeline!
> 
> - Brian Gesiak
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

On 2020-01-29, Reid Kleckner via llvm-dev wrote:
>On Wed, Jan 29, 2020 at 2:27 AM Serge Guelton via llvm-dev <
>llvm-dev at lists.llvm.org> wrote:
>
>> My personal take on this would be to start moving forward. Still
>> supporting both
>> version this year, but obsoleting Python 2.7 and requiring, say Python
3.6,
>> starting January 2021 looks like a good compromise.
>>
>
>Sounds good to me. Keeping the window of time during which we support both
>Python 2 and 3 as small as possible would be nice.
One blocker may be shebang, #!/usr/bin/env python

may pick either Python 2 or Python 3, if the distributions haven't
migrated.

Some scripts has dome
`#!/usr/bin/env python` -> `#!/usr/bin/env python3` but some have not.
If we can update the rest, it may be easier for our overall migration.

I also recall Reid said `#!/usr/bin/env python` might make Windows
developers happier but I forget the details.
>_______________________________________________
>LLVM Developers mailing list
>llvm-dev at lists.llvm.org
>https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev