Python 3 support for building Swift Toolchain

Hey all-

Fedora is actively deprecating Python 2 for Python 3 and as of just a few days ago, the change was made that /usr/bin/python is no longer symlinked to /usr/bin/python2 but /usr/bin/python3. I rebuilt the latest 2019-07-14 snapshot and it fails with, you guessed it, Python errors.

As Python 2 is soon to be EOL'ed (as noted by this amusing site) I was wondering if there was any "official" plan to upgrade the build scripts to Python 3. I was thinking to do it and apply a set of patches to make Swift build on Fedora, but am happy to contribute them back as pull requests. That said, I don't want to duplicate others work, so if nobody's actively working on it, mind if I go ahead and do it?

Thanks,

Ron

16 Likes

Okay, I am able to build Swift with Python 3. Pull requests are here and here. I mentioned that it's probably early to be submitting the pull request insofar as I'm guessing the automated builders are still 2.7; I wanted to make the code available for whenever it's decided to move forward with Python 3.

3 Likes

As a followup, I have a Swift 5.1-2019-07-24 rpm available here for Fedora that was built on a Python 2-free environment; python3 had to be explicitly symlinked to python because that's what build-script expects to find. :upside_down_face:

1 Like

Hey @Ron_Olson, I'd like to thank you for pushing this effort forward. It will likely simplify packaging for many different distributions. In particular, I know that the Arch Linux swift-language package on AUR does a fair bit of regex replacing to patch the build script to explicitly use Python 2. I decided to try out your changes by building swift-DEVELOPMENT-SNAPSHOT-2019-07-22-a, and they have worked smoothly so far, minus one exception—I had to patch import diagnostics in utils/swift_build_support/swift_build_support/host_specific_configuration.py.

Unfortunately, however, I ran into an error a while into the build. I'm not actually certain if this error is related to the Python version, so if it's not, I'll file a bug report. Linked here are some pertinent lines from the crash. The two lines that seem most suspect are (firstly) FAILED: docs/CMakeFiles/docs_html after Building HTML documentation and (once it begins building the Sphinx documentation) Warning, treated as error: /home/aemino/aur/swift-language/src/swift/docs/ARCOptimization.rst:100:Pygments lexer name 'sil' is not known

The following warning:
/home/aemino/aur/swift-language/src/swift/utils/line-directive:709: FutureWarning: Possible nested set at position 36 '(?P<middle>, line |:)(?P<line>[0-9]+)(?P<tail>.*?)\n?$')
also shows up a fair bit.

I have very little knowledge about Swift's build process, so your input would be appreciated here. Thanks!

1 Like

Yep, I'm very familiar with that error. The issue is that somewhere the python-sphinx tool, using pygments, can't find the custom SIL colorizer; I looked into it a bit and I think the idea is that the build script wants to pass it to pygments as it's not on the known list of colorizers, and it fails. That was going to be next on the list of things to investigate.

In the meantime, the solution is straightforward: remove python-sphinx from your system; the Swift toolchains builds fine without it. I updated the Fedora spec file and removed python-sphinx as a requirement.

The FutureWarning is the thing I'm looking into now; it's not an error as you mention but it would be nice to be as warning-free as possible.

FYI there's a great article on gyb that really helped explain how it's being used as part of the build process; once I read the article I "got it" in terms of what is trying to be accomplished, and actually found that it's a separate project unto itself and have found my new boilerplate-generator tool. :grin:

2 Likes

Glad to hear it's not just me! I'll remove python-sphinx and retry the build tonight. Thanks for your help and good luck fixing the issue!

FYI, I found the problem, but it turns out it was already fixed via this pull request., so it shouldn't be an issue at some point soon (it's still there in the 2019-07-25 build).

Just in case you haven't seen it already, Gwynne did a lot of work in https://github.com/apple/swift/pull/23256 and I did some pieces (some repeated) in https://github.com/apple/swift/pull/23038 (sadly mixed with other changes).

Oh, wow, cool, no, I didn't know about these pull requests. Awesome work!

It'll definitely be interesting to see how the 2->3 transition happens with macOS; it's been the source of a lot of headaches in Fedora (at least for me :upside_down_face:).

Two months to go till the Python Software Foundation drops support for Python 2. I notice that Ron's pulls sit unmerged. Can someone update on what the plan is here?

4 Likes

So it's now officially been EOL'd, and I'm unable to build locally due to python pointing to a python 3 after brew install python. I can hack around it for now, but it seems like a good time to at least update the "Getting Started" to reflect that utils/build-script is not expected to work.

1 Like

I'm currently updating the patch so Swift 5.2 will build on Fedora.

I'm assuming that Python 3 will be supported by Apple soon insofar as it's been announced that the next version of macOS, whatever it's called, will no longer have built-in support for scripting languages (Python, Ruby, etc.) so when people go get their own version of Python, it's likely gonna be a 3.x version. That said, I have no actual idea when/if that will happen.

What is the function of Python in the Swift toolchain? Is there any chance this could be deprecated in the future in favor of something which is less fussy to deal with?

1 Like

Python is quite integral to building Swift; it acts as a glue between other components, actually generates Swift code via GYB, and LLDB, integral to the Swift REPL, provides a scripting API for it.

Personally, I doubt Python will be replaced anytime soon, if ever, in the Swift ecosystem.

It's no longer just Arch, Fedora and other bleeding edge distributions without support for Python 2. We can add Ubuntu 20.04 to that growing list: With Python 2 EOL'ed, Ubuntu 20.04 LTS Moves Along With Its Python 2 Removal, Python2 removal for focal.

I know that 20.04 isn't even in beta yet but it's also not far away either (April 2nd). I think it's fair to start asking what a plan for addressing this looks like. There are proposals on the table that I think merit discussion/review.

3 Likes

I have gone ahead and written an article that explains how to build a tool chain for Swift 5.2 on Ubuntu 20.04. On some level, I think it's fair to say that this is how you build the Swift 5.2 tool chain on any Linux distribution without Python 2 (but that is an untested conjecture). If you try it and something is unclear/missing/wrong please let me know.

The tl;dr is there are 6 patches for two projects that need to be applied.

  1. apple/swift#30662
  2. 0001-Updated-to-work-with-Python-3-swift-5.2.patch
  3. 0002-Change-CMake-Python-detection-swift-5.2.patch
  4. 0003-Potpourri-fixes-to-make-compilation-work.patch
  5. apple/swift-integration-tests#61
  6. 0001-Update-SourceKit-LSP-to-use-Python-3.patch

Most of the patches listed above are changes @Ron_Olson has suggested in one form or another. Patch #2 is a modified version of @Ron_Olson's patch in apple/swift#26296. The changes in patch #4 to host_specific_configuration.py probably should be included in whatever follow up to apple/swift#26296 Ron makes. Patch #6 is the comment I left on swift-integration-tests#61.

Patch #3 I think should be turned into a merge request and added to mainline. The only problem that I see with it is that it bumps the minimum required CMake version to 3.12.4 (from 3.4.3). There's likely a way to work around that with conditional code if that was a problem.

The patch to build-presets.ini in #4 is weird and I don't understand it. :man_shrugging:

I think my next goal is to get an Ubuntu 20.04 node added to the Swift Community-Hosted Continuous Integration.

3 Likes

I feel like I am close to having a complete build that works on Python 2 and 3 (GYB and all tests). I keep getting stuck on something and I just do not know enough Python to actually solve the problem. I am asking for some help. This line in PathSanitizingFileCheck is giving me fits.

As it currently is in master if run on Python 3 I get: TypeError: a bytes-like object is required, not 'str'.

So I changed it to:

stdout, stderr = p.communicate(stdin.encode("utf-8"))

This solves the Python 3 error but then on Python 2 I get: UnicodeEncodeError: 'utf-8' codec can't encode characters in position 56928-56930: surrogates not allowed.

Is there a Python 2/3 compatible way of solving this?

You should probably pass an encoding= to the Popen initialiser on the previous line instead.

Thank you I will give that a try; I will let you know. It'll take 2+ hours for it to run through my CI matrix. :laughing:

Unfortunately that did not work, TypeError: __init__() got an unexpected keyword argument 'encoding', looking at the Popen constructor for Python 2.7 that seems right. There is no encoding parameter.