Fedora is actively deprecating Python 2 for Python 3 and as of just a few days ago, the change was made that /usr/bin/python is no longer symlinked to /usr/bin/python2 but /usr/bin/python3. I rebuilt the latest 2019-07-14 snapshot and it fails with, you guessed it, Python errors.
As Python 2 is soon to be EOL'ed (as noted by this amusing site) I was wondering if there was any "official" plan to upgrade the build scripts to Python 3. I was thinking to do it and apply a set of patches to make Swift build on Fedora, but am happy to contribute them back as pull requests. That said, I don't want to duplicate others work, so if nobody's actively working on it, mind if I go ahead and do it?
Okay, I am able to build Swift with Python 3. Pull requests are here and here. I mentioned that it's probably early to be submitting the pull request insofar as I'm guessing the automated builders are still 2.7; I wanted to make the code available for whenever it's decided to move forward with Python 3.
As a followup, I have a Swift 5.1-2019-07-24 rpm available here for Fedora that was built on a Python 2-free environment; python3 had to be explicitly symlinked to python because that's what build-script expects to find.
Hey @Ron_Olson, I'd like to thank you for pushing this effort forward. It will likely simplify packaging for many different distributions. In particular, I know that the Arch Linux swift-language package on AUR does a fair bit of regex replacing to patch the build script to explicitly use Python 2. I decided to try out your changes by building swift-DEVELOPMENT-SNAPSHOT-2019-07-22-a, and they have worked smoothly so far, minus one exception—I had to patch import diagnostics in utils/swift_build_support/swift_build_support/host_specific_configuration.py.
Unfortunately, however, I ran into an error a while into the build. I'm not actually certain if this error is related to the Python version, so if it's not, I'll file a bug report. Linked here are some pertinent lines from the crash. The two lines that seem most suspect are (firstly) FAILED: docs/CMakeFiles/docs_html after Building HTML documentation and (once it begins building the Sphinx documentation) Warning, treated as error: /home/aemino/aur/swift-language/src/swift/docs/ARCOptimization.rst:100:Pygments lexer name 'sil' is not known
The following warning: /home/aemino/aur/swift-language/src/swift/utils/line-directive:709: FutureWarning: Possible nested set at position 36 '(?P<middle>, line |:)(?P<line>[0-9]+)(?P<tail>.*?)\n?$')
also shows up a fair bit.
I have very little knowledge about Swift's build process, so your input would be appreciated here. Thanks!
Yep, I'm very familiar with that error. The issue is that somewhere the python-sphinx tool, using pygments, can't find the custom SIL colorizer; I looked into it a bit and I think the idea is that the build script wants to pass it to pygments as it's not on the known list of colorizers, and it fails. That was going to be next on the list of things to investigate.
In the meantime, the solution is straightforward: remove python-sphinx from your system; the Swift toolchains builds fine without it. I updated the Fedora spec file and removed python-sphinx as a requirement.
The FutureWarning is the thing I'm looking into now; it's not an error as you mention but it would be nice to be as warning-free as possible.
FYI there's a great article on gyb that really helped explain how it's being used as part of the build process; once I read the article I "got it" in terms of what is trying to be accomplished, and actually found that it's a separate project unto itself and have found my new boilerplate-generator tool.
FYI, I found the problem, but it turns out it was already fixed via this pull request., so it shouldn't be an issue at some point soon (it's still there in the 2019-07-25 build).
Oh, wow, cool, no, I didn't know about these pull requests. Awesome work!
It'll definitely be interesting to see how the 2->3 transition happens with macOS; it's been the source of a lot of headaches in Fedora (at least for me ).
Two months to go till the Python Software Foundation drops support for Python 2. I notice that Ron's pulls sit unmerged. Can someone update on what the plan is here?
So it's now officially been EOL'd, and I'm unable to build locally due to python pointing to a python 3 after brew install python. I can hack around it for now, but it seems like a good time to at least update the "Getting Started" to reflect that utils/build-script is not expected to work.
I'm currently updating the patch so Swift 5.2 will build on Fedora.
I'm assuming that Python 3 will be supported by Apple soon insofar as it's been announced that the next version of macOS, whatever it's called, will no longer have built-in support for scripting languages (Python, Ruby, etc.) so when people go get their own version of Python, it's likely gonna be a 3.x version. That said, I have no actual idea when/if that will happen.
What is the function of Python in the Swift toolchain? Is there any chance this could be deprecated in the future in favor of something which is less fussy to deal with?
Python is quite integral to building Swift; it acts as a glue between other components, actually generates Swift code via GYB, and LLDB, integral to the Swift REPL, provides a scripting API for it.
Personally, I doubt Python will be replaced anytime soon, if ever, in the Swift ecosystem.
I have gone ahead and written an article that explains how to build a tool chain for Swift 5.2 on Ubuntu 20.04. On some level, I think it's fair to say that this is how you build the Swift 5.2 tool chain on any Linux distribution without Python 2 (but that is an untested conjecture). If you try it and something is unclear/missing/wrong please let me know.
The tl;dr is there are 6 patches for two projects that need to be applied.
Most of the patches listed above are changes @Ron_Olson has suggested in one form or another. Patch #2 is a modified version of @Ron_Olson's patch in apple/swift#26296. The changes in patch #4 to host_specific_configuration.py probably should be included in whatever follow up to apple/swift#26296 Ron makes. Patch #6 is the comment I left on swift-integration-tests#61.
Patch #3 I think should be turned into a merge request and added to mainline. The only problem that I see with it is that it bumps the minimum required CMake version to 3.12.4 (from 3.4.3). There's likely a way to work around that with conditional code if that was a problem.
The patch to build-presets.ini in #4 is weird and I don't understand it.
I feel like I am close to having a complete build that works on Python 2 and 3 (GYB and all tests). I keep getting stuck on something and I just do not know enough Python to actually solve the problem. I am asking for some help. This line in PathSanitizingFileCheck is giving me fits.
As it currently is in master if run on Python 3 I get: TypeError: a bytes-like object is required, not 'str'.
This solves the Python 3 error but then on Python 2 I get: UnicodeEncodeError: 'utf-8' codec can't encode characters in position 56928-56930: surrogates not allowed.
Is there a Python 2/3 compatible way of solving this?
Unfortunately that did not work, TypeError: __init__() got an unexpected keyword argument 'encoding', looking at the Popen constructor for Python 2.7 that seems right. There is no encoding parameter.
