[Pitch] Bump Python's required version from 3.9 to 3.11

Development LLDB
1

I would like to propose that the required version of Python for LLDB be bumped from 3.9 to 3.11 or higher.

Motivation

When installing the ARM64 Swift toolchain on Windows, users have to install Python 3.9 ARM64. However, the official Python website does not provide an installer for Python 3.9 ARM64. Therefore, a user installing the Swift toolchain on Windows ARM64 would either have to build Python3.9 from source or install from a third party like Nuget.

The first release of Python for which the official website provides an ARM64 installer is 3.11. Bumping the required version of Python to 3.11 would improve the installation experience for ARM64 Windows users, as they would be able to download the Python installer from the official website.

Blockers

I am not sure what is involved in bumping the required Python version for the toolchain.

cc @JDevlieghere @mishal_shah @compnerd

3 Likes
2

And/or maybe bundle the Python installer with the Swift toolchain installer (as it's only ~23 MB)?

4 Likes
3

Adding an option in the swift GUI installer to install Python 3.9.10 from a bundled version would work and is probably simpler than bumping the minimum required version.

It would also fix #79981 for good.

4

I’m more interested in keeping at the same version of Xcode. If Xcode is bumping to 3.11 then windows would follow suite. Adding support to install python from the installer is interesting, and something that we could do as a post step (that is not bundle the installer, but the URL). One of my hopes is to convert to an online installer by default - it would make the installer exe ~1 MB, so 23MB doesn’t work. I’m happy to review patches for an option to install python.

1 Like
5

We have some precedent when it comes to different platforms supporting different versions of Python. At least for Ubuntu, we use whichever Python version is the default for that release. Upstream we have pretty good test coverage for the different Python versions, with some bots running 3.9 (notably the macOS one) and the Linux ones running 3.11 and 3.12.

Personally, I think the lack of an ARM64 installer is sufficiently strong motivation to bump the Windows toolchain to a newer version of Python.

3 Likes
6

TODOs to bump the Python version on Windows

Here is the list of things I can think of that we would have to change in order to update the python version:

  1. Change the Python URLs in build.ps1.
  2. Update the documentation on the Swift website which instructs to install Python 3.9.x on Windows.

I don't think we need to update anything on the Windows ci.swift.org bots, as the Python version is handled by build.ps1, which handles the script.

Consequences on users

One consequence of this change on users is that those who update their toolchain would see the missing python311.dll error message when they run lldb for the first time (if they did not read the changelog). I think the message is clear enough to help them fix the issue.

Adding the Python URL to the Swift installer

The Pythons .nupkg available on Nuget are not installers, they are a zip file of the Python toolchains. In build.ps1 we decompress them in the correct directory to install them.

If we were to install Python through the Swift installer, then we would have to manually edit the PATH and some other logic that the Python installer already handles. At a first glance, this seems to add a lot of complexity to the installer.

We could also build the 3.9.10 installer for ARM64 (if that is supported), host it and add that URL to the Swift installer instead of the Nuget one.

Overall this approach seems more complex than updating the Python version in the docs and build.ps1.

7

I would rather that we don't diverge from macOS versions on Windows. I think that ARM64 not having the installers is not a big deal - Python has supported Windows ARM64 since 3.9 - we have been building against it since we added support for the platform.

This is a non-change - the same thing occurs with Python39.dll instead.

You wouldn't add the URL you would chain the installer. You should look up how burn does chained installers. This is actually exactly what we are doing with the toolchain today. We chain the toolchain, IDE tools, debugging tools, CLI tools, and the SDKs. This isn't too far removed from today's means of installation, but it does add additional complexity as we would need to include a bit of UI to handle the chaining post installation like other installers do.

1 Like
8

The link says it is no longer maintained and 3.11 is recommended.

Then it is maybe time for Xcode to do this bump :wink:

1 Like
9

Sure, a newer python would be reasonable in the future.

I do not have the authority or influence for that - but if Xcode were to do so, I would try to make the Windows releases match the selected version.

1 Like
10

But we do diverge on Linux? Is that what I read above?

Having lived this for the last year working on SwiftPM for Windows it is a horrible user experience trying to get a python that works, at least for those of us using ARM Windows trying to target all our platforms from a single device.

11

I also use Windows on ARM64 devices (albeit not ASi). I would be open to someone extending the installer to (optionally) chain the Python installation into the Swift toolchain installer. I don't think that requires diverging on the version here.

12

But we do diverge on Linux? Is that what I read above?

If you look at the package bots:
https://ci.swift.org/job/oss-swift-package-ubuntu-20_04/4328/consoleText
https://ci.swift.org/job/oss-swift-package-ubuntu-24_04/1422/consoleText

we, for example, build LLDB on Ubuntu 20.04 against 3.8.10 and Ubuntu 24.04 against 3.12.3.

13

If we were to go through effort of chaining the installers, wouldn't it make more sense to do that for a newer Python version that is still supported?

14

I don't think that the version selection would be an issue there - the cost is in the building of the infrastructure, not the version selection.

15

@Adrian_Prantl and I discussed this offline.

I think that we came to a compromise that seems to possibly resolve the issue in a supportable way.

  1. we need to get CI coverage for LLDB on Windows on both llvm-project and swift
  2. we would want to continue to ensure that the python version are the same across amd64 and arm64 Windows
  3. we would want to ideally chain the installation via the installer.

The official LLVM releases use Python 3.10, and so it is preferable that we use Python 3.10 as well if we bump. Fortunately, python.org has a embedded python for ARM64 and AMD64 at 3.10.1.

We could repackage the embedded python as a MSI and add that as an optional component in the installer. This would add an additional ~10MB to the installer and would be an opt-in for the user.

Going forward then, we would tie ourselves to the python version that llvm.org uses on the stable branch that we build from.

3 Likes
16

For completeness, I just wanted to note that historically our (swift.org) stable branches are not necessarily in sync with any LLVM stable branches. But I understand the desire to use a Python version that has gotten some living on in LLDB outside of the Swift community.

1 Like
17

Actually, we did switch to tracking stable points for the rebranch. The next sync point is release/21.x.

18

This issue doesn’t seem to be very Swift specific as it would also affect non-swift windows lldb users, it might be worth pitching bumping the python required version in the llvm discourse directly.

Also have we considered the implications of diverging both the upstream version and the swift fork ? I’m concerned it might introduce python code divergence as well, because of deprecated APIs on newer versions of python for instance.

1 Like
19

This is basically my concern as well. I think that the bump to 3.10 should be relatively safe as that is what upstream is currently using on Windows.

1 Like