llvm dev - Jan 2021 - [RFC] Python 2 / Python 3 status, final step(s)

Nevermind. I went ahead and took the risk of installing Python 3 and
putting it on PATH, and the buildbot service code seems to be working as
is. Both bots have made it past cmake, so they're probably on track to
recover.

On Thu, Dec 17, 2020 at 11:38 AM Reid Kleckner <rnk at google.com> wrote:
> Can we delay this? I maintain two Windows buildbots, and I'm about to
go
> on holiday break, and I'd really rather not scramble to get them up to
date:
> http://lab.llvm.org:8011/#/builders/clang-x64-windows-msvc
> http://lab.llvm.org:8011/#/builders/sanitizer-windows
>
> Looking at them now, it seems you broke them five hours ago.
>
> There is risk involved in simply installing Python 3, since it might
> necessitate upgrading the buildbot worker version, which I have
> successfully avoided doing for almost five years now.
>
> This has not been a good month for Windows bots. As a long suffering
> maintainer, I'm asking for a reprieve.
>
> On Mon, Dec 14, 2020 at 10:30 PM Serge Guelton via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hi Folks,
>>
>> unless someone has strong arguments against it, the transition to
Python
>> 3 will happen *this week*, on Thursday morning, european time.
>> The associated review is https://reviews.llvm.org/D93097, feel free to
>> drop in.
>> The impact should be low, Python3 is already picked by CMake as
default,
>> but a Python 2 fallback still exists, and it will be removed.
>>
>> **If you're a bot owner** please make sure Python3 is available on
your
>> system ! Now, is the time :-)
>>
>> On Mon, Oct 19, 2020 at 9:34 PM Jonas Devlieghere via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> Sounds good. In LLDB we have a slightly different timeline for the
test
>>> suite (outlined in
>>> http://lists.llvm.org/pipermail/lldb-dev/2020-August/016388.html)
but
>>> everything else aligns with the plan here. I already removed the
fallback
>>> to Python 2 in preparation for the 12.0 release.
>>>
>>> On Tue, Oct 13, 2020 at 11:49 AM David Blaikie via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> *thumbs up*
>>>>
>>>> Re: out of date buildbots: maybe stage the shebang changes in a
>>>> time/way that's easy to rollback (or even intended to
rollback after a
>>>> small time window (few hours) off-peak) to flush out the
buildbots. Then
>>>> after that rollout, go over the buildbot failures and follow-up
with
>>>> buildbot owners to get them cleaned up before a more
intended-to-stick
>>>> rollout is done.
>>>>
>>>> On Tue, Oct 13, 2020 at 7:18 AM James Y Knight via llvm-dev
<
>>>> llvm-dev at lists.llvm.org> wrote:
>>>>
>>>>> Yay, thanks for persevering with this migration!
>>>>>
>>>>> On Tue, Oct 13, 2020 at 8:04 AM Serge Guelton via llvm-dev
<
>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>
>>>>>> Hi Folks,
>>>>>>
>>>>>> Now that LLVM 11.0.0 has been released, it's time
to prepare for the
>>>>>> final step
>>>>>> envisionned in the previous RFC named *[RFC] Python 2 /
Python 3
>>>>>> status* [0],
>>>>>> ie. requiring Python3.6 for LLVM 12.0.0, to be released
in 2021.
>>>>>> At least Fedora already only ships Python3 and we
didn't have much
>>>>>> bugs reported
>>>>>> wrt. Python compatibility for the LLVM toolchain.
>>>>>>
>>>>>> Indeed, all Python scripts should now be at least
compatible with
>>>>>> both python2 (py2)
>>>>>> and python3 (py3), some of them already are already py3
only.
>>>>>>
>>>>>> The build system still depends on Python in a few
place, but it
>>>>>> explicitly
>>>>>> mentions Python3_EXECUTABLE, and the main dependency
(llvm-build) is
>>>>>> currently
>>>>>> being removed  in D89142.
>>>>>>
>>>>>> The shebangs have already been harmonized in D83857:
some mention
>>>>>> /usr/bin/env
>>>>>> python, some mention /usr/bin/env python3, and none
mention python2
>>>>>> anymore. It
>>>>>> would be great to have all script use the same shebang,
PEP394 [1]
>>>>>> makes it
>>>>>> (relatively) clear that in our case, explicitly
requiring python3 is
>>>>>> the way to
>>>>>> go.
>>>>>>
>>>>>> So basically, next step would be to update the
documentation, remove
>>>>>> the Python2
>>>>>> fallback in root CMakeLists.txt and update shebangs.
>>>>>>
>>>>>> Anyone seeing an issue with that approach? I'm
still a bit worried
>>>>>> about
>>>>>> buildbots still running on py2 (?)
>>>>>>
>>>>>> Serge
>>>>>>
>>>>>> [0]
http://lists.llvm.org/pipermail/llvm-dev/2020-January/138730.html
>>>>>> [1] https://www.python.org/dev/peps/pep-0394/
>>>>>>
>>>>>> _______________________________________________
>>>>>> LLVM Developers mailing list
>>>>>> llvm-dev at lists.llvm.org
>>>>>>
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>
>>>>> _______________________________________________
>>>>> LLVM Developers mailing list
>>>>> llvm-dev at lists.llvm.org
>>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20201217/6c2d93b5/attachment.html>

Thanks Reid for taking this risky step. As someone who just lost one day
upgrading/downgrading/messing up its laptop, I fully appreciate the risk
you took, and I'm glad everything looks fine.

The two slaves I had issues whith were:

    http://lab.llvm.org:8011/#/builders/127 (now green)
    http://lab.llvm.org:8011/#/workers/14 (now green)

Thanks a bunch!
> As a long suffering maintainer
And extra thanks for the maintenance task too o/

On Thu, Dec 17, 2020 at 9:19 PM Reid Kleckner <rnk at google.com> wrote:
> Nevermind. I went ahead and took the risk of installing Python 3 and
> putting it on PATH, and the buildbot service code seems to be working as
> is. Both bots have made it past cmake, so they're probably on track to
> recover.
>
> On Thu, Dec 17, 2020 at 11:38 AM Reid Kleckner <rnk at google.com>
wrote:
>
>> Can we delay this? I maintain two Windows buildbots, and I'm about
to go
>> on holiday break, and I'd really rather not scramble to get them up
to date:
>> http://lab.llvm.org:8011/#/builders/clang-x64-windows-msvc
>> http://lab.llvm.org:8011/#/builders/sanitizer-windows
>>
>> Looking at them now, it seems you broke them five hours ago.
>>
>> There is risk involved in simply installing Python 3, since it might
>> necessitate upgrading the buildbot worker version, which I have
>> successfully avoided doing for almost five years now.
>>
>> This has not been a good month for Windows bots. As a long suffering
>> maintainer, I'm asking for a reprieve.
>>
>> On Mon, Dec 14, 2020 at 10:30 PM Serge Guelton via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> Hi Folks,
>>>
>>> unless someone has strong arguments against it, the transition to
Python
>>> 3 will happen *this week*, on Thursday morning, european time.
>>> The associated review is https://reviews.llvm.org/D93097, feel free
to
>>> drop in.
>>> The impact should be low, Python3 is already picked by CMake as
default,
>>> but a Python 2 fallback still exists, and it will be removed.
>>>
>>> **If you're a bot owner** please make sure Python3 is available
on your
>>> system ! Now, is the time :-)
>>>
>>> On Mon, Oct 19, 2020 at 9:34 PM Jonas Devlieghere via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> Sounds good. In LLDB we have a slightly different timeline for
the test
>>>> suite (outlined in
>>>>
http://lists.llvm.org/pipermail/lldb-dev/2020-August/016388.html) but
>>>> everything else aligns with the plan here. I already removed
the fallback
>>>> to Python 2 in preparation for the 12.0 release.
>>>>
>>>> On Tue, Oct 13, 2020 at 11:49 AM David Blaikie via llvm-dev
<
>>>> llvm-dev at lists.llvm.org> wrote:
>>>>
>>>>> *thumbs up*
>>>>>
>>>>> Re: out of date buildbots: maybe stage the shebang changes
in a
>>>>> time/way that's easy to rollback (or even intended to
rollback after a
>>>>> small time window (few hours) off-peak) to flush out the
buildbots. Then
>>>>> after that rollout, go over the buildbot failures and
follow-up with
>>>>> buildbot owners to get them cleaned up before a more
intended-to-stick
>>>>> rollout is done.
>>>>>
>>>>> On Tue, Oct 13, 2020 at 7:18 AM James Y Knight via llvm-dev
<
>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>
>>>>>> Yay, thanks for persevering with this migration!
>>>>>>
>>>>>> On Tue, Oct 13, 2020 at 8:04 AM Serge Guelton via
llvm-dev <
>>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>>
>>>>>>> Hi Folks,
>>>>>>>
>>>>>>> Now that LLVM 11.0.0 has been released, it's
time to prepare for the
>>>>>>> final step
>>>>>>> envisionned in the previous RFC named *[RFC] Python
2 / Python 3
>>>>>>> status* [0],
>>>>>>> ie. requiring Python3.6 for LLVM 12.0.0, to be
released in 2021.
>>>>>>> At least Fedora already only ships Python3 and we
didn't have much
>>>>>>> bugs reported
>>>>>>> wrt. Python compatibility for the LLVM toolchain.
>>>>>>>
>>>>>>> Indeed, all Python scripts should now be at least
compatible with
>>>>>>> both python2 (py2)
>>>>>>> and python3 (py3), some of them already are already
py3 only.
>>>>>>>
>>>>>>> The build system still depends on Python in a few
place, but it
>>>>>>> explicitly
>>>>>>> mentions Python3_EXECUTABLE, and the main
dependency (llvm-build) is
>>>>>>> currently
>>>>>>> being removed  in D89142.
>>>>>>>
>>>>>>> The shebangs have already been harmonized in
D83857: some mention
>>>>>>> /usr/bin/env
>>>>>>> python, some mention /usr/bin/env python3, and none
mention python2
>>>>>>> anymore. It
>>>>>>> would be great to have all script use the same
shebang, PEP394 [1]
>>>>>>> makes it
>>>>>>> (relatively) clear that in our case, explicitly
requiring python3 is
>>>>>>> the way to
>>>>>>> go.
>>>>>>>
>>>>>>> So basically, next step would be to update the
documentation, remove
>>>>>>> the Python2
>>>>>>> fallback in root CMakeLists.txt and update
shebangs.
>>>>>>>
>>>>>>> Anyone seeing an issue with that approach? I'm
still a bit worried
>>>>>>> about
>>>>>>> buildbots still running on py2 (?)
>>>>>>>
>>>>>>> Serge
>>>>>>>
>>>>>>> [0]
>>>>>>>
http://lists.llvm.org/pipermail/llvm-dev/2020-January/138730.html
>>>>>>> [1] https://www.python.org/dev/peps/pep-0394/
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> LLVM Developers mailing list
>>>>>>> llvm-dev at lists.llvm.org
>>>>>>>
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>>
>>>>>> _______________________________________________
>>>>>> LLVM Developers mailing list
>>>>>> llvm-dev at lists.llvm.org
>>>>>>
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>
>>>>> _______________________________________________
>>>>> LLVM Developers mailing list
>>>>> llvm-dev at lists.llvm.org
>>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20201218/408b4710/attachment.html>

Hi folks,
>From my experience with running the mlir-nvidia band clangd buildbot
workers and setting up pre-merge testing I can really recommend setting up
the workers using Docker (or a similar technology):

   - easy versioning/diffing/reverting of your changes
   - you can run two different images in production and staging
   - Easy to copy-and-past to run a different config
   - You can test your exact changes locally, before deploying them. This
   gives you much faster iteration times.

Downsides:

   - Need to get familiar with Docker
   - Need to workaround infrastructure issues if you have some special
   usages (e.g. GPU access)
   - Bumpy experience on Windows

My Dockerfiles are in llvm-zorg
<https://github.com/llvm/llvm-zorg/tree/main/buildbot/google/docker>, feel
free to reuse these.


Best,
Christian


On Fri, Dec 18, 2020 at 9:11 AM Serge Guelton via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Thanks Reid for taking this risky step. As someone who just lost one day
> upgrading/downgrading/messing up its laptop, I fully appreciate the risk
> you took, and I'm glad everything looks fine.
>
> The two slaves I had issues whith were:
>
>     http://lab.llvm.org:8011/#/builders/127 (now green)
>     http://lab.llvm.org:8011/#/workers/14 (now green)
>
> Thanks a bunch!
>
> > As a long suffering maintainer
>
> And extra thanks for the maintenance task too o/
>
> On Thu, Dec 17, 2020 at 9:19 PM Reid Kleckner <rnk at google.com>
wrote:
>
>> Nevermind. I went ahead and took the risk of installing Python 3 and
>> putting it on PATH, and the buildbot service code seems to be working
as
>> is. Both bots have made it past cmake, so they're probably on track
to
>> recover.
>>
>> On Thu, Dec 17, 2020 at 11:38 AM Reid Kleckner <rnk at
google.com> wrote:
>>
>>> Can we delay this? I maintain two Windows buildbots, and I'm
about to go
>>> on holiday break, and I'd really rather not scramble to get
them up to date:
>>> http://lab.llvm.org:8011/#/builders/clang-x64-windows-msvc
>>> http://lab.llvm.org:8011/#/builders/sanitizer-windows
>>>
>>> Looking at them now, it seems you broke them five hours ago.
>>>
>>> There is risk involved in simply installing Python 3, since it
might
>>> necessitate upgrading the buildbot worker version, which I have
>>> successfully avoided doing for almost five years now.
>>>
>>> This has not been a good month for Windows bots. As a long
suffering
>>> maintainer, I'm asking for a reprieve.
>>>
>>> On Mon, Dec 14, 2020 at 10:30 PM Serge Guelton via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> Hi Folks,
>>>>
>>>> unless someone has strong arguments against it, the transition
to
>>>> Python 3 will happen *this week*, on Thursday morning, european
time.
>>>> The associated review is https://reviews.llvm.org/D93097, feel
free to
>>>> drop in.
>>>> The impact should be low, Python3 is already picked by CMake as
>>>> default, but a Python 2 fallback still exists, and it will be
removed.
>>>>
>>>> **If you're a bot owner** please make sure Python3 is
available on your
>>>> system ! Now, is the time :-)
>>>>
>>>> On Mon, Oct 19, 2020 at 9:34 PM Jonas Devlieghere via llvm-dev
<
>>>> llvm-dev at lists.llvm.org> wrote:
>>>>
>>>>> Sounds good. In LLDB we have a slightly different timeline
for the
>>>>> test suite (outlined in
>>>>>
http://lists.llvm.org/pipermail/lldb-dev/2020-August/016388.html) but
>>>>> everything else aligns with the plan here. I already
removed the fallback
>>>>> to Python 2 in preparation for the 12.0 release.
>>>>>
>>>>> On Tue, Oct 13, 2020 at 11:49 AM David Blaikie via llvm-dev
<
>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>
>>>>>> *thumbs up*
>>>>>>
>>>>>> Re: out of date buildbots: maybe stage the shebang
changes in a
>>>>>> time/way that's easy to rollback (or even intended
to rollback after a
>>>>>> small time window (few hours) off-peak) to flush out
the buildbots. Then
>>>>>> after that rollout, go over the buildbot failures and
follow-up with
>>>>>> buildbot owners to get them cleaned up before a more
intended-to-stick
>>>>>> rollout is done.
>>>>>>
>>>>>> On Tue, Oct 13, 2020 at 7:18 AM James Y Knight via
llvm-dev <
>>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>>
>>>>>>> Yay, thanks for persevering with this migration!
>>>>>>>
>>>>>>> On Tue, Oct 13, 2020 at 8:04 AM Serge Guelton via
llvm-dev <
>>>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>>>
>>>>>>>> Hi Folks,
>>>>>>>>
>>>>>>>> Now that LLVM 11.0.0 has been released,
it's time to prepare for
>>>>>>>> the final step
>>>>>>>> envisionned in the previous RFC named *[RFC]
Python 2 / Python 3
>>>>>>>> status* [0],
>>>>>>>> ie. requiring Python3.6 for LLVM 12.0.0, to be
released in 2021.
>>>>>>>> At least Fedora already only ships Python3 and
we didn't have much
>>>>>>>> bugs reported
>>>>>>>> wrt. Python compatibility for the LLVM
toolchain.
>>>>>>>>
>>>>>>>> Indeed, all Python scripts should now be at
least compatible with
>>>>>>>> both python2 (py2)
>>>>>>>> and python3 (py3), some of them already are
already py3 only.
>>>>>>>>
>>>>>>>> The build system still depends on Python in a
few place, but it
>>>>>>>> explicitly
>>>>>>>> mentions Python3_EXECUTABLE, and the main
dependency (llvm-build)
>>>>>>>> is currently
>>>>>>>> being removed  in D89142.
>>>>>>>>
>>>>>>>> The shebangs have already been harmonized in
D83857: some mention
>>>>>>>> /usr/bin/env
>>>>>>>> python, some mention /usr/bin/env python3, and
none mention python2
>>>>>>>> anymore. It
>>>>>>>> would be great to have all script use the same
shebang, PEP394 [1]
>>>>>>>> makes it
>>>>>>>> (relatively) clear that in our case, explicitly
requiring python3
>>>>>>>> is the way to
>>>>>>>> go.
>>>>>>>>
>>>>>>>> So basically, next step would be to update the
documentation,
>>>>>>>> remove the Python2
>>>>>>>> fallback in root CMakeLists.txt and update
shebangs.
>>>>>>>>
>>>>>>>> Anyone seeing an issue with that approach?
I'm still a bit worried
>>>>>>>> about
>>>>>>>> buildbots still running on py2 (?)
>>>>>>>>
>>>>>>>> Serge
>>>>>>>>
>>>>>>>> [0]
>>>>>>>>
http://lists.llvm.org/pipermail/llvm-dev/2020-January/138730.html
>>>>>>>> [1] https://www.python.org/dev/peps/pep-0394/
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> LLVM Developers mailing list
>>>>>>>> llvm-dev at lists.llvm.org
>>>>>>>>
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> LLVM Developers mailing list
>>>>>>> llvm-dev at lists.llvm.org
>>>>>>>
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>>
>>>>>> _______________________________________________
>>>>>> LLVM Developers mailing list
>>>>>> llvm-dev at lists.llvm.org
>>>>>>
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>
>>>>> _______________________________________________
>>>>> LLVM Developers mailing list
>>>>> llvm-dev at lists.llvm.org
>>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210105/04af5e08/attachment-0001.html>