This proposal introduces the ability to rely on a dependency statically (meaning that it will be available for import in your Package.swift file). The idea is that this will greatly increase the usability and customization of the Package.swift file.
Read more: SE-NNNN spm-static-deps.md
Thanks for writing the proposal Tanner!
I am in favor of this proposal but I have some concerns that I want to call out:
It might obfuscate the manifest file. Usually, a user would want to get complete information about a package from its manifest file but overuse of this feature could might make it complex and difficult to understand a package. That said, it is already possible to use Swift code in manifest file and you can import Foundation and do all sorts of stuff. So, this may not be a very serious problem in practice.
It will definitely make manifest loading slower. Loading Swift package manifest is slow because we need to invoke the compiler. Adding more compilation steps and Swift modules will further slow down the loading process.
We plan to solve 2. by caching the manifest files. We might not be able to implement proper caching if the package model is changed outside of the actual manifest file.
Thanks Ankit!
Totally agree. This was my initial concern as well. However, the solution to the problem of preventing obfuscation would seem to be requiring a Package.json type format where arbitrary code execution is impossible and the structure is guaranteed to be predictable and intelligible. Given SPM didn't go that route, my thoughts are we might as well take advantage of having a .swift file.
This should only affect packages that make use of static dependencies at least.
Currently, you can load in some arbitrary .json files, make network requests, ask for user input, etc before declaring the var package: Package. Would it make a difference if that happens within the Package.swift file itself or as the result of an imported method call?
I completely understand the desire of the proposal.
The part that I struggle with is embedding specific meaning into comments that have to be parsed and then some "magic" happens to make certain parts of the Package.swift have the ability to run different code.
This seems like a big deviation away from the core rationale for using Swift to begin with. This essentially creates a mini-DSL within the comments that only work for SwiftPM Swift code and not regular Swift code. So I struggle with how to defend that decision.
I wonder if a plug-in architecture wouldn't be more appropriate, albeit it would be more work. With the bootstrapping problem, you'd almost need a Package-Plugins.swift and then a Package.swift. Definitely not ideal, but at least we're back to using standard Swift again.
SwiftPM executes the manifest in a sandbox (on macOS), so things like network requests are not allowed. We also don't guarantee when or how many times the manifest will be executed. Loading manifests is a huge performance hit and we want to aggressively cache everything. This proposal will make that very difficult I think.
Trying to aggressively cache the output of an arbitrary Swift script seems like fighting a losing battle. Even if network requests aren't allowed (it seems like they still are on Linux), you can get input from many other places. In other words, if the goal of SPM was to be fast it should not have chosen .swift for the manifest format.
If SPM is going to have a .swift package manifest, we must take advantage of it. The .swift format has a ton of tradeoffs in terms of tooling. Most importantly, developers cannot reliably parse, edit, or create SPM manifests programatically. If the benefit of actually having Swift in these manifests is being undermined by optimization requirements, then it's really not worth it to be using Swift there.
Perhaps this is worth another proposal, but an alternative solution to the tooling problem presented in the above proposal is for SPM to support a subset of manifest functionality via a Package.json file (that would be used in-place of a Package.swift). This obviously wouldn't support everything you can do in the .swift version of the manifest, but it would work great for the majority of use cases and be much friendlier to developer tooling. Essentially the motivation behind this thread's proposal is to empower developers to implement this Package.json file themselves.
I updated the proposal with a couple of new sections.
Without the ability to rely on other modules, the "Swift" being used in Package.swift is effectively a DSL--it's not the full Swift.
Without the ability to rely on other modules, the “Swift” being used in Package.swift is effectively a DSL–it’s not the full Swift.
Yep, I totally agree. But, like you mentioned above, if we're just going to create a textual representation of those dependencies in the comments, a YAML or JSON file is probably a better option over the Package.swift-DSL + the Package.swift+StaticLibComments-DSL, or some other external swift-tool implementation to add and remove data from the Package.swift file.
I guess what I was asking above is, would you be opposed to a solution that provided a PackageConfig.swift (or whatever name) that defines the set of imports you're trying to bring it (e.g. MySPMUtils)?
This makes SwiftPM have a two-step process, which is a perf hit, but allows for the Package.swift file to essentially become normal Swift again. It does add some additional complexity to the mix, of course.
I'm not opposed to the idea, but I do see a couple of problems with it:
1: Declaring dependencies is a very small subset of what the Package.swift actually does. There's no need to declare products, targets, etc when declaring static dependencies. You simply need to declare an array of dependencies. That means it would probably look something like this:
import PackageDescription
let config = PackageConfig(dependencies: [
.package(url: "https://github.com/tanner0101/my-spm-util.git", "1.0.0"..<"2.0.0")
])
The plus side here is we can re-use the Version and Package Dependency structs from PackageDescription, but really there's not a big use case for the functionality of Swift here. More importantly, the next problem arises...
2: Would you at some point want static dependencies for your static dependency manifest? Having one Swift manifest file that supports static dependencies and one that doesn't is confusing.
I do definitely agree that the swift-tools comment format is non-ideal. A different solution to this would be something like an spm.config (or SPMConfig.json, etc) file that contains some information that should be parsed prior to parsing the Package.swift file.
However, given that the SPM team opted for a comment solution to the versioning problem (swift-tools-version) which is also a before-parsing-this-file configuration step, it would seem best to follow-suit there and keep things consistent.
I tend to agree with what @owensd has mentioned on this topic regarding both comments and the general desire for plugins (or some other mechanism by which SPM can be extended), and the idea that this definitely creates a major paradigm shift in how the build tools work. The biggest and most obvious concern I have with the proposal as-is is the idea that the PackageDescription models (which are supposed to be there for community integration with these tools) would no longer be valid unless you also implemented a bunch of comment-parsing and linking logic.
I remember @ddunbar (possibly? it was quite a while ago) bringing up the idea that SwiftPM itself would be using a sort of "self-hosted plugin architecture" -- which is an interesting idea that opens up several doors, including some use-cases that probably are motivated by the same motivation as this proposal -- but if it breaks the compile-time safety of the Package spec as this proposal currently does, I think it's likely better to explore other implementation options.
Adding a bit to describe better what I meant by "self-hosted plugin architecture" -- this would entail an approach that describes dependencies as just a simple data structure (let's call it "workspace.json" for the purposes of this conversation):
{
"build": {
"SwiftPMTools": "4.0.x",
"https://path/to/MyFancyTestFrameworkBuilder": "x.x.x"
},
"runtime": {
"https://path/to/afnetworking-or-something": "x.x.x",
},
"local": {
"https://path/to/MyFancyTestFrameworkRuntime": "x.x.x"
}
}
Notice that you'd have to define a specific version of SwiftPMTools.
In this case, SwiftPM parses workspace.json and invokes (for example) project.swift with all the configured build dependencies. And if you want to dump out a package that's parseable and "importable" to other packages, you can do that too. This has a few benefits:
With build tools like this it's often a winning bet to design the system to be extensible and get out of the community's way. Of course I'm sure this represents major regressions with the Xcode integration, but I wonder how wise it is to let a closed-source, single-platform text editor / build-tool define what a cross-platform open-source build-tool can and can't do.
I have concerns about the proposal, but I want to focus on one point that you bring up:
Developers can hack around this by using Regular Expressions or other pseudo-Swift parsing methods, but nothing that works reliably. There is simply no way to do this in the current version of SPM. To do this reliably, you would need to parse the Swift into an AST, modify that AST, and serialize the Swift. But even that incredibly involved solution still has a lot of unanswered problems.
At the moment, libSyntax is incomplete, but in the timeframe in which this proposal would be implemented I assume that will no longer be the case. At that point, I think having a more limited DSL makes quite a bit of sense from a tooling perspective - it means you can focus on a core subset of Swift when using some hypothetical libSyntax-based CLI to edit package definitions.
In that case I think it makes sense to expose that limited functionality as a runtime API to allow other tools to allow the community to build upon it, as using something like libSyntax to do those transformations has the drawback of requiring actual Swift sources as an intermediary.
A multi-step compilation process like what I described above has better project-locality (the dependencies are managed by SwiftPM inside the project, rather than externally/globally as I'd imagine some tool that edits my Package.swift would be installed), and therefore, better backwards-compatibility (you can install an older runtime library of the tools on a single legacy project without having to support it in every future release).
Maybe I'm getting far enough afield on these responses that it's worth another thread?
That's another point I don't understand in this debate: Requiring a centralized format like a (whitelisted) Swift file to build other Swift code, requiring it not be Turing Complete, requiring you stay within a well-supported ecosystem - why are these limiting, why are they undesirable? When I'm not writing code, I'm fighting three different build systems and maintaining a linear number of manifest files for them across a dozen frameworks already. I shudder at the thought of being allowed to delegate responsibility to yet more external formats or libraries Just Because YAML.
My package manager should not be able to launch nukes. It should build code.
While I agree with that in principle, I think there's a certain amount of tension between "SwiftPM the package manager" and "SwiftPM the build tool." There are people who want the build tool to do things for their dev environment that a "package manager" shouldn't do -- and the lack of the ability of the package manager to define alternative local dependencies stifles many alternative workflows. So I'm offering an option that reduces the pain of moving to an alternative setup, one that also tries to reduce the churn for people who don't need that flexibility.
But more than anything it takes that complexity off of a build-tool and makes it more opt-in for users.
In NodeJS, I can choose a test framework (qUnit, Jasmine, Mocha, among others), a bundler (Browserify, Webpack, Rollup, among others), and I can import and manage a pile of test-only and runtime dependencies, which means if I want a separate task-runner for my project, I just add gulp or grunt as a dev dependency.
In Java, I can install maven, gradle, or ant plugins. I can define different "profiles" in which those libraries are exposed. In Clojure, Leiningen plugins. In Ruby, I import code into a Rake task and call it.
Why can't I do anything remotely close that with SwiftPM? I have to use XCTest (which has a horribly limited user experience on Linux) or SwiftPM won't play. My test dependencies, if I even bother to try and put them into Package.swift, are exposed to consumers of my package, which means that if two libraries depend on differing versions of a test dependency, anyone who consumes both is in trouble.
So while I agree with the idea that a build-tool shouldn't be needlessly complex, I at least want to recognize that the problem-space itself is very complex, what with all the potential use-cases, and I think offloading that complexity to a generic process (resolve dependencies, run build file linking build-specific dependencies) is easier and offers less code-churn than any alternative I've yet seen, other than "let me specify local dependencies available in test environment, or testTargets which aren't XCTest bundles" which frustratingly also isn't supported. this is kinda supported with targets that aren't included in products, but even that exposes the dependencies of those targets to consumers
it's not "Because YAML," it's "Because I Don't See Swift As Solely The iOS App Language And Want It To Offer Similar Workflows And Use-Cases That Are Well-Worn In Languages It May Be Competing With It Like Node, Python, Ruby, and Others"
EDIT: Either of these approaches works. One jives with what was discussed in a Slack Q&A quite a while back and that may no longer be valid -- been a busy few months and haven't been able to weigh in, so if I'm out-of-date, please correct me!
I completely understand your frustration with lack of these features in SwiftPM. We really want to add first class support for things you listed, its just that we haven't gotten around to doing them. The Swift community should feel free to start discussions with a draft proposal about these features!
Thanks for the reply Ankit! And yeah, I think that (drafting a proposal) is probably overdue for me.
But I want to make clear my expression of frustration is meant in terms of "let's try to drive the community to make some real decisions, here", rather than "let's get our pitchforks out and drive the SwiftPM team out of here" -- I appreciate all the hard work y'all are doing!
Just to be clear, Xcode is not part of the Swift open source project and in no way drives SwiftPM design decisions.
Just to be clear, Xcode is not part of the Swift open source project and in no way drives SwiftPM design decisions.
I'd agree w/ this philosophy in general, so that's reassuring... thanks Ankit. The reason I even brought it up is that last I heard, testDependencies was removed from Package due to not playing nice with the XCTest bundles on macOS (for some reason that I didn't quite understand) -- but that may be outdated or inaccurate information, considering the reason for its removal was over my head in the first place?
If there's a concrete path forward on enabling test/local dependencies in some way, I'd try to help at taking a stab at an implementation, FWIW. I'd probably need some coaching and/or code-review, but if you've got time to complain, you've got time to code ;)