[Pitch] Swift Projects

Evolution Pitches
1

Background

After SPM became integrated with Xcode, Apple and others (including some contributors to the Swift.org forums) began recommending Swift Packages not only for distributing source code, but also, for local development use—creating modules as libraries and later, frameworks—a fine alternative to Xcode project files.

Today, some of us even use Swift Packages solely in this latter capacity—we only use packages as a replacement for much-hated Xcode project files, but we have yet to start using SPM to manage our remote dependencies.

Example Use Case

For example, at my org, we use Carthage to build our third-party dependencies as XCFrameworks. Our tried-and-true cacheing solution, Rome, adapted easily to these. We eventually want to switch to using SPM for our dependencies, but that's another topic.

Meanwhile we now have upwards of 20 different Swift Packages throughout our workspace, which each define one or more separate, local modules (which we would have otherwise created using Xcode Project files).

The Current Problem

So, what's the problem with using SPM to create local modules?

We have run into a multitude of bugs and difficulties when trying to use Swift Packages as a replacement for .xcproj files in our project. SPM was designed to facilitate distributing dependency libraries—not as a functional equivalent to .xcproj files.

Here's a short list of the issues:

  • Xcode refuses to compile any local Xcode projects that link to a Swift package whose Package.swift contains "unsafeFlags" (workaround is an ugly hack involving a wrapper package without unsafe flags)
  • Under certain circumstances, Xcode tries to statically links dynamic frameworks, resulting a build failure due to duplication of symbols
  • Despite an earlier proposal claiming to add a way to link binary dependencies to a Swift package, there is still no good way to do this, because Xcode does't prioritize building binaryTargets before it tries to build the source packages that depend on them.
  • For each dynamic target declared in a package, Xcode builds a static library in $(BUILT_PRODUCTS_DIR) and a dynamic framework in $(BUILT_PRODUCTS_DIR)/PackageFrameworks, which causes all kinds of havok if you use $(BUILT_PRODUCTS_DIR) as a Framework Search Path.
  • There is no good way to specify particular build options just for iOS simulator builds.
  • Swift Packages cannot link to (import) targets that are declared in an .xcproj file.

The Solution (General)

To address this, rather than continuing to try to shoehorn SPM into a role that it was never designed to fill, instead, we should make a new, similar solution called Swift Projects.

Swift Projects would share part of the same syntax as a Swift Package, making it easy to migrate existing SPM packages to being Projects. Setting up a basic module (framework or library) would be done the same way.

However, there'd be no facility at all for a Swift Prioject to reference a separate git repo or manage versions in a complex way. At most, there could be a version tag, but it would function identically to how version info is represented in Xcode projects. all the use-cases of Xcode project

Another difference: Swift Projects would have auto-complete-friendly parameters with doc comments for each and every compiler flag and build option that's currently available to set in the Xcode Project File interface.

Most critically: Swift Projects would carry the guarantee that the result of running xcodebuild on a Project target would be 100% identical to running the same xcodebuild on a framework with identical build options (many of which are currently only achievable via arcane unsafe flags).

The one exception to this "identicalness" would be file handling. Files in Swift Projects would be handled like Swift Packages handles files: whatever is in a folder, is in the darn folder. No more "group without a folder" or file references with UUIDs etc,

Thus it should be technically feasible to convert any Xcode Project File to a Swift Project, but there might be some symlinks created to make a file appear in multiple places.

Lastly, Swift Projects should play nice with playgrounds, and should be able to link to any prebuilt (cached) binary framework or static library.

The Solution (Specific)

I haven't really gotten as far as writing up what the exact syntax should look like. I'm hoping that's what this thread might discuss :D

At a later time I will link to all the various problem threads related to SPM, if anyone is interested.

6 Likes
2

Even if we had this it wouldn't do what you want, which is to fix Xcode. Unfortunately that's up to Apple and that doesn't seem likely to change any time soon. What might be useful here is to identify the capabilities or changes package definitions would need in order to fix these issues, beyond what Xcode does. If certain things seem beyond what SPM should provide, then start thinking about some sort of intermediate format.

5 Likes
3

What might be useful here is to identify the capabilities or changes package definitions would need in order to fix these issues, beyond what Xcode does. If certain things seem beyond what SPM should provide, then start thinking about some sort of intermediate format.

Apple's the one who started pushing the idea of using Swift Packages in place of Xcode project files for local use.

I don't know which of these issues and problems is due to Xcode and which is due to SPM. All I know is that I want to be able to press a button, and have it convert any Xcode project directly into a Swift package, and have it just work.

Currently, we are very far away from something like this due to all the issues.

Here is a short list of some of the problems:

Issues with Unsafe Flags

  • Currently, Xcode projects will fail to build if they directly link to a Swift Package that uses any unsafe flags. The only known workaround (other than just using an Xcode Project instead of a Swift Package) is to make another Swift Package that doesn't use unsafe flags, and have it import the one that does use unsafe flags, then in this "wrapper" package have a single .swift file that has @_exported import UnsafePackage and then link the wrapper to the other Xcode projects.

Issues related to linking external frameworks to a Swift Package NOT using .binaryTarget:

Issues related to .binaryTarget

Issues related to weak linking:

Issues related to headers:

Issues related to dynamic library package targets being treated as static:

Issues related to both the static linking thing and XCFrameworks:

Bugs related to Sanitizers

Bugs related to Resources

  • A resources bundle that's inside a Swift Package cannot be directly accessed by consumers.
    • This seems to be a known issue mentioned by Apple in the WWDC talk about using resources in a Swift package.
    • The workaround is for the package to provide public accessor methods that provide the resources from the bundle to consumers.
    • However, because you cannot have mixed Swift/Obj.C SPM targets, and because of the above issue where @objc code in a Swift SPM target can't be used by actual Obj. C code in modules declared in Xcode projects, no SPM target can provide resources to both Objective C code and to Swift code, making it not really worthwhile to try migrating Xcode projects that vend resources to being Swift packages, and causing us a lot of wasted time in trying but failing to do so.
  • Swift Package with resources causing test targets not to compile
  • SPM resources in test use wrong bundle path (duplicate of above?)

Issues related to Unsafe Flags

  • Xcode refuses to compile any local Xcode projects that link to a Swift package whose Package.swift contains "unsafeFlags" (workaround is an ugly hack involving a wrapper package without unsafe flags)

Performance issues with Xcode

  • package resolution always runs when you open Xcode, taking sometimes over a minute
  • In Xcode 12.5, the initial step of pasting in the link to Google Firebase package to add it as a dependency incurs a 10–15 minute delay before the next screen loads (which shows which various libraries you can add as dependencies)
    • This seems to be due to the fact that SPM performs a deep clone of all the related repos before it lets you do anything else, which for a repo like Firebase is frickin' huge
  • In Xcode 12.5, initially loading a project that has Firebase as a package dependency requires nearly 20 minutes for package resolution to succceed due the deep cloning of all the dependency repos
  • Xcode does not support loading a project without performing package resolution even if your Derived Data folder already contains all the build products of all the packages depended on by your app and frameworks

Various other issues with the SPM/Xcode integration

  • For each dynamic target declared in a package, Xcode builds a static library in (BUILT_PRODUCTS_DIR) and a dynamic framework in (BUILT_PRODUCTS_DIR)/PackageFrameworks, which causes all kinds of havok if you use $(BUILT_PRODUCTS_DIR) as a Framework Search Path.
  • There is no good way to specify particular build options solely for iOS simulator builds.
  • Swift Packages cannot link to (import) targets that are declared in an .xcproj file.
  • lack of support for mixed source targets
  • confusing Package manifest
    • how are we supposed to know which build setting goes in swiftSettings and which settings go in cSettings and which settings go in linkerSettings?
    • How are we supposed to know what counts as an unsafe flag and what can be defined?
    • Why isn't there just a Config struct that has all the possible settings already defined as variables with doc comments that you can simply change the value of, similar to how you fill out that big spreadsheet of build settings for Xcode projects?
  • editing a package manifest for 2 seconds causes it to recompile, often even if you didn't press "save"
    • if you haven't finished editing it yet this will cause your currently selected build device in Xcode to switch to Mac instead of iOS Simulator even if the Package manifest explicitly only supports iOS. This is supremely annoying!
  • SPM does not support custom config names in a coherent manner -- it just has "Debug" and "Release".
    • If your Xcode project uses any custom config names, SPM uses a secret, hidden "heuristic" to decide whether to use its Debug or Release config based on the custom name of your config. This "heuristic" tends to pick the wrong thing—for example if you have a config named "Testing", SPM will build the "Release" config instead of "Debug" and thus not actually build the package with testability enabled!
  • If you use SPM's .linkedFramework build option to link a target to another build product, Xcode's Find Implicit Dependencies fails to realize that this linkedFramework has to be built first before trying to build the package, and so your build can fail intermittently unless you turn off "Find Implicit Dependencies" and "Parallelize Build" and explicitly specify a build order that guarantees the package won't be built until the linked framework has been built.
    • This issue is extraordinarily frustrating
3 Likes
4

And that's not to even mention all the performance issues related to SPM:

  • not using shallow clones to checkout dependency repos
  • lack of support for mixed source targets
  • confusing Package manifest—how are we supposed to know which build setting goes in swiftSettings and which settings go in cSettings and which settings go in linkerSettings? How are we supposed to know what counts as an unsafe flag and what can be defined? Why isn't there just a Config struct that has all the possible settings already defined as variables with doc comments that you can simply change the value of, similar to how you fill out that big spreadsheet of build settings for Xcode projects?
  • editing a package manifest for 2 seconds causes it to recompile, which (if you haven't finished editing it yet) will cause your currently selected build target in Xcode to switch to Mac instead of, e.g., iOS Simulator, even if the Package manifest explicitly only supports iOS
  • no supported way to designate build settings specific to simulator builds as opposed to device builds, forcing us to use extremely ugly hacks to accomplish this
  • package resolution always runs when you open Xcode, taking sometimes over a minute
  • the method to add a package dependency to a framework is completely different if the package is locally defined vs. remotely defined
  • no way to define custom config names besides "Debug" and "Release", and if you do, SPM uses a hidden "heuristic" to decide which of these to use (and it usually picks the wrong thing; if you have a config named "Testing" for example, it will build the "Release" config instead of "Debug" and thus not build anything for testing)
  • no way to use a shared .xcconfig file (that I'm aware of)
  • no support for directly linking to binaries like Xcode projects can, unless you link to it as a target which is based on the assumption of distributing a Package with some pre-compiled target that's part of the distribution—so it doesn't support simply linking a local Package to a pre-compiled framework or library in a shared folder outside the directory structure of the package)
  • if you use the .linkedFramework option to link to another build product, Xcode's Find Implicit Dependencies fails to realize that this linkedFramework has to be built first before trying to build the package, and so your build can fail intermittently

It started to be come a full-time job for me to create sample projects for, and report, each and every one of these issue, only to see Xcode 12.5 sit on beta 3 for a whole month without any movement on the issues that were still in the build related to SPM.

I'm this close " to throwing in the towel on using SPM instead of Xcode projects for local stuff, because we want to be able to use TSAN and now we can't because we followed Apple's encouragement to use SPM for local projects. I realize it will take Apple some time to address all of these issues but I honestly don't know which of these is due to Xcode and which of these is due to the fact that SPM wasn't designed as a replacement for Xcode projects and so why should we be surprised that it's not well-suited to this?

Maybe it would make more sense to start from scratch and design something from the ground up only to be a replacement for Xcode projects. That way we could decouple and totally separate the versioning and distribution concerns of SPM from the build setting and target definition concerns of projects.

3 Likes
5

Some additional problems we've run into since I wrote the above message:

  • @objc APIs declared in a Swift package cannot be used by Objective C code in a consuming module that's declared via an Xcode project because SPM/Xcode fails to add the appropriate Headers folder to the framework it builds. Swift forum link (bug report link to bugs.swift.org is provided in that post).
  • A resources bundle that's inside a Swift Package cannot be directly accessed by consumers. This seems to be a known issue mentioned by Apple in the WWDC talk about using resources in a Swift package. The workaround is for the package to provide public accessor methods that provide the resources from the bundle to consumers. However, because you cannot have mixed Swift/Obj.C SPM targets, and because of the above issue where @objc code in a Swift SPM target can't be used by actual Obj. C code in modules declared in Xcode projects, no SPM target can provide resources to both Objective C code and to Swift code, making it not really worthwhile to try migrating Xcode projects that vend resources to being Swift packages, costing us a lot of wasted time in trying but failing to do so.
3 Likes