Is a semantic version of the SwiftSyntax package for Swift 5.1 coming soon? @akyrtzi
While I wait, I have been toying with a fork in an attempt to get a head start on updating client code, but I have been running into trouble. I couldn’t find a swift-5.1-RELEASE tag, so I checked out swift-5.1-branch. Then I de‐GYB‐ified the package. In that state, it builds fine, and the tests pass in the package manager or with the result of generate-xcodeproj. However, if I open the package directly in Xcode by double‐clicking Package.swift, the built product (and anything that depends on it) is inoperable. The errors are all variations of this:
xctest (30953) encountered an error (Failed to load the test bundle. (Underlying error: The bundle “SwiftSyntaxTest” couldn’t be loaded because it is damaged or missing necessary resources. The bundle is damaged or missing necessary resources. dlopen_preflight([...]/Library/Developer/Xcode/DerivedData/swift-syntax-glexjgjummeedhaqfonfiovodnqm/Build/Products/Debug/SwiftSyntaxTest.xctest/Contents/MacOS/SwiftSyntaxTest): Library not loaded: @rpath/lib_InternalSwiftSyntaxParser.dylib
Referenced from: [...]/Library/Developer/Xcode/DerivedData/swift-syntax-glexjgjummeedhaqfonfiovodnqm/Build/Products/Debug/SwiftSyntaxTest.xctest/Contents/MacOS/SwiftSyntaxTest
Reason: image not found))
The same errors are inherited by client packages.
What is _InternalSwiftSyntaxParser, and where is it supposed to come from?
I wonder if this is something that can be fixed or worked around in the tagged release?
Until the SwiftSyntax team tags an official release, you can do what swift-format is doing and depend on the 2019-07-10 development snapshot; there weren't any incompatible changes to the syntax tree definitions between that and the final Swift 5.1 release.
I think this is a bug (?) in Xcode where it's not putting one of the toolchain's library directories in the binary's rpath correctly. I don't know if there's a better way to workaround it, but I just end up putting a symlink to lib_InternalSwiftSyntaxParser.dylib in the same directory as the executable so it'll find it, and that sticks around unless I clean the build product directory.
Aliasing lib_InternalSwiftSyntaxParser.dylib is sufficient to make Xcode work. I will probably stick with generate-xcodeproj for now for simplicity’s sake, but it is good to know it can be made to work and that nothing more serious is wrong.
However, once I got that “fixed”, I moved to the next step: Linux. There (both Ubuntu 16 and 18) my client tests consistently crashed when SwiftSyntax tried to unwrap the dlsym(_:_:) result in AtomicCounter.swift. I solved it by “breaking the rules” for _CSwiftSyntax, giving it a header and module map, and importing it normally. Whatever the issue was, it vanished.
How important is the “self-contained Swift module” nature of SwiftSyntax when it comes to the semantic version tags intended for consumption as a package? The rationale listed in the import directory is this:
Don't put headers here to import via the _CSwiftSyntax module. Swift does not support private module imports, meaning when distributing the SwiftSyntax.module we would also have to bundle these private headers as well.
But the distribution of headers seems unimportant in the case where everything is already managed automatically by the package manager. Would it be reasonable to include the extra headers alongside the GYB output just for those particular tags?
The dlsym issue has been fixed in master by this commit, which makes the _CSwiftSyntax module an @_implementationOnly import and thus no longer "leaks" it out of the module. Once they tag a release for 5.1, they should also pick up that change.
Thanks for bringing up these issues, I've opened a PR to fix the linux issue for 5.1.
A thing to keep in mind is that the SwiftSyntax library only depends on lib_InternalSwiftSyntaxParser.dylib being present and that library is self-contained, you can copy it out of the toolchain (e.g. if you have an app using SwiftSyntax you can bundle both SwiftSyntax and parser library within the app to make it relocatable).
Also note that SwiftSyntax and parser library are generally version locked (e.g. 5.1 SwiftSyntax will require the 5.1 parser library) and if the loaded parser library is incompatible, SwiftSyntax will throw a parsing error.
I'm trying to use SwiftSyntax on iOS 13 (Simulator) using Xcode 11.0 (11A420a) -- AppStore
I added SwiftSyntax using Xcode's GUI, and it indicates the version is 0.50100.0 as you commented just over a week ago.
-- What's the right tag to use for SwiftSyntax to get exactly the version in my current Xcode? I'm getting the following error: Could not find or use auto-linked library '_InternalSwiftSyntaxParser'
To be clear: I'm not trying to Build Swift from Source, and my hello world-esque usage of SwiftSyntax successfully compiles, it just doesn't link!
Is it a fools errand to be attempting to get this to work on iOS? (I don't know if all the dependencies are able to compile on iOS at all!)
Essentially you'd need to build the parser library for simulator (and device). The parser library inside the Xcode toolchain is only built for macOS.
If you checkout the swift-5.1-branch sources you can copy the build-parser-lib script from master and invoke it as I mentioned in the other forum thread.
Thanks for the reply!! a primary goal of mine in today's experiments was to prove that it could work at all on iOS. -- I hadn't run across your other thread before now.
I am trying to use the Swift Package Manager to build a macOS command line tool to play with the SwiftSyntax library. But currently I am running into a runtime error "dyld: Library not loaded: @rpath/lib_InternalSwiftSyntaxParser.dylib". I am using Xcode 11.2.1 and pointing SPM at the master branch of SwiftSyntax and tag "0.50100.0" as per the readme.
I see from this thread and others that the dyld error appears to be a known problem. One suggestion is to "put a symlink to lib_InternalSwiftSyntaxParser.dylib in the same directory as the executable" but I am not sure where to find lib_InternalSwiftSyntaxParser.dylib, is that part of the main Swift branch?
Some other comments in this thread indicate that the dyld issue has been fixed in the swift-5.1-branch branch but that branch is now a couple of months old and has not been merged into master. If I point SPM at swift-5.1-branch there are build errors which seem to stem from the fact that the .gyp files have not been processed. Is there a way to add this step as part of my build process?
I apologize in advance if I have missed something obvious and thanks for the help.
I am trying to use the Swift Package Manager to build a macOS command line tool to play with the SwiftSyntax library. [...] I am using Xcode 11.2.1 and pointing SPM at the master branch of SwiftSyntax and tag "0.50100.0" as per the readme.
Then everything should already work when you do swift build, swift run, etc. from the command line. It should also work when you do do swift package generate-xcodeproj and then use that generated project.
But currently I am running into a runtime error "dyld: Library not loaded: @rpath/lib_InternalSwiftSyntaxParser.dylib".
You should only run into this if you open the package itself in Xcode by double‐clicking Package.swift. I simply don’t use it that way, because the other two ways are simpler. But if you need to do this, then you need to symlink the library from the corresponding toolchain. You will want to create a script or something that automates it, because it will vanish every time you clean the build directory.
I am not sure where to find lib_InternalSwiftSyntaxParser.dylib
lib_InternalSwiftSyntaxParser.dylib is in the Swift toolchain. With a basic Xcode install from the App Store, that would be /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/macosx/lib_InternalSwiftSyntaxParser.dylib.
Some other comments in this thread indicate that the dyld issue has been fixed in the swift-5.1-branch branch
Ignore that. The very first comment that mentioned the branch is out of date; everything the branch had then is also in 0.50100.0. The last few posts brought the branch up again, but they are talking about something else (building from source for iOS).
This is the scenario in which the "dyld: Library not loaded: @rpath/lib_InternalSwiftSyntaxParser.dylib" error appears during runtime of the tool. Adding a link to lib_InternalSwiftSyntaxParser.dylib next to the binary fixes this issue. I put the creation of the link into a build script as you suggested.
Since it sounds like using the build system without Xcode does not have this issue perhaps there is a bug in Xcode's SPM support?
You could also work around it in your case by doing this:
Check out the SwiftSyntax repository at 0.50100.0 in a neighbouring directory, as a submodule, or any similar method. (Or do it with the top‐level package if SwiftSyntax is actually a transitive dependency of a different package.)
Run swift package generate-xcodeproj once in SwiftSyntax’s repository.
Drag that Xcode project into your workspace.
There are two reasons I suggest this over the symlink strategy:
It will work consistently no matter where Swift happens to be installed, whereas the symlink and its script will break if you check your project out on a machine where Swift is installed elsewhere.
All parts of the workaround live in the repository; nothing is in the products directory. That means you can set it up once and it will work indefinitely. The symlink on the other hand needs to be put in place repeatedly every time.
Here's a hacky workaround. You can run this after the first build for testing, then running the tests from within Xcode should work.
for path in ~/Library/Developer/Xcode/DerivedData/*/Build/Products/Debug/; do
cp "$(xcode-select -p)/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/macosx/lib_InternalSwiftSyntaxParser.dylib" $path
done
For a long time I have just used swift run and swift test instead as on Linux. The issue only manifests at runtime, so I still use Xcode for editing source files.
Just wondering why this is the solution? Surely Swift's own syntax library should be usable without manually embedding a library from the toolchain. This issue complicates the usage of a variety of tools that build atop of SwiftSyntax.
