Package Manager Workspace

I think we can easily expand this feature in future to use a Swift file and the JSON option can remain for simpler use cases. This proposal aims to solve problems that community is facing right now. Using Swift file is more consistent and powerful but it comes with a lot of complexity.

2 Likes

We shouldn't need to worry about what term Xcode uses as SwiftPM is a separate software (also, Xcode is not part of Swift OSS project). That said, I don't strongly feel what we call this feature and I am open to naming suggestions.

1 Like

The overall idea sounds great, but the inconsistency of using JSON instead of the existing Swift format strikes me as a something that’ll cause dramas down the road.

I’d want to write the package/workspace descriptions in the same format.

7 Likes

Thanks for the feedback Tony! I am not sure what you mean by drama. I updated the proposal to explain why I think we should use JSON for now (and then move to a Swift based format in future).

1 Like

Sorry, yes, I’ll clarify: once the JSON format is deployed, rolling it back could take a long time. It also would likely encourage people to ask for the same for standard package manifests.

I know it’s not the easier technical solution, but I feel that using a consistent format for both package and workspace manifests is a better experience for developers/users of SPM.

5 Likes

I don't think supporting a subset of options will be a technical burden so we might not necessarily need to roll it back instantly. However, if it turns out that no one wants to use the JSON file because everyone has moved to the Swift based format, we can easily deprecate and remove it.

I completely agree that using a Swift based format will provide a consistent and better experience in future. I think the point I am trying to make is that we don't lose anything by having a JSON based format.

It does feel like it adds churn, if we think it should ultimately be Swift to be consistent. I tend to agree with Tony here...

3 Likes

Moving to Swift based file will also depend on how this feature evolves over time and with what complexity. I don't think it is always good to implement something which provides way less flexibility just to be consistent with other parts.

1 Like

I agree with @Aciid here, JSON is so easy to parse, doesn't need any additional tooling and Swift workspace definitions can be added later. I find the fact that pure package manifests can't be declared in JSON or some other easily parsed language very inconvenient. If consistency is desired, adding a possibility to have JSON package manifest would work much better.

1 Like

The term "workspace" is already widely used when working with other package managers, where it has the same meaning, check out Workspaces | Yarn

1 Like

I get that, but my sense (from other discussions I've read too) is that there seems to be a bit of underlying tension between Swift and non-Swift solutions - both here and even for the package format itself.

I think that tension needs resolving.

If you go for JSON here for pragmatic reasons, it somewhat undercuts the arguments for sticking with Swift for the package. I'm hoping that there will be plans sooner rather than later to expand the scope of the package manifest to support settings and more sophisticated build operations. They are also problems that the community is facing right now, and an argument could be made for a solution to them that just involves JSON too, for similar reasons (it's simpler, we can get it up and running faster, etc).

I'm not arguing for one or the other (I can see pros & cons both ways), I'm just arguing for consistency and the sense that there's a long term plan that's being followed, rather than a series of short-term decisions which might leave us with a bit of an incoherent picture over time.

7 Likes

I agree there are problems that needs to be addressed in other parts of the package manager. This feature is orthogonal to other outstanding issues and missing features. This proposal does not aim to solve or indirectly influence other parts of the package manager.

Right. We definitely don't want to put ourselves in a position that we cannot recover from. I've already mentioned how this feature can evolve overtime and move away from JSON in a clean way.

1 Like

I had more discussions about this proposal and its clear that Swift is much more preferable and consistent. I completely dropped JSON based manifest from the proposal and introduced a Swift based manifest.

PS: I am still thinking if we actually need the userInfo property in the Swift based manifest.

3 Likes

I have difficulties to understand what workspace is meaning to work.
At first I though it was about defining custom package dependency overload (of Package.swift) while developing a package. But at the end you say:

It will not be allowed to have both Package.swift and Workspace.swift
in the same directory. We may lift this restriction in future by adding an
option to prefer one over the other.

So I'm confused. What's the role of Workspace? How does it integrate with existing SPM and especially Package.swift if we can't have both of them?

Same for me. I don't get the objective of this proposal, for what would be used? It seems it has the same objective as the Local Dependencies proposal.

I chatted with @NeoNacho, @ddunbar, @rballard and @abertelrud this morning in person about this proposal. We think it may be possible to compose the functionality proposed here using the local dependencies feature. However, there are several open questions:

• This proposal allows cloning packages automatically but there is no such functionality in the local dependencies. The local dependency feature can be extended to include a URL or people can write scripts to perform the cloning operations.

• Users will need to create a target and declare dependency on the products they want to build. This may be a good thing because it allows control over what is actually built. We can add a special case: if there are no targets and all dependencies are local, we build all available products. Or, we can add a flag to SwiftPM to build everything.

• It was also mentioned that people may want a global edit mode for the packages they usually work on. This can be done by a global swiftpm config file, similar to gitconfig. Since this config could be highly personal, there is a concern around sharing that within a team.

One thing to consider: how does the workspace build system handle external library dependencies, especially when there might be conflicts?

For example, consider a workspace with 3 packages: MyFoundation, FooKit, and BarKit.

MyFoundation depends on an external package CoolUtils from [1.0,2.0). FooKit depends on CoolUtils [1.0, 1.1) and BarKit depends on CoolUtils [1.1, 2.0). FooKit and BarKit both depend on MyFoundation, but they do not depend on each other. In the simplest implementation of workspaces this set of packages is simply impossible to build as the combination of dependencies is unsatisfiable. There is a way to satisfy them if we build MyFoundation twice, once against CoolUtils 1.1 and once against CoolUtils 1.0:

CoolUtils 1.1 <- MyFoundation <- BarKit

CoolUtils 1.0 <- MyFoundation <- FooKit

Ankit, do you have any thoughts as to how we could represent this with this proposal?

It is not possible to have different version of the same module in Swift. Your example sounds like dependency hell situation. swift-package-manager/README.md at f303078b2fd2f58e6ab9973442dbde79edfd3e58 · apple/swift-package-manager · GitHub

But the workspace I described is perfectly valid when FooKit and BarKit are developed as separate packages - the problem only arises when I want to place them into the same workspace. If FooKit was actually FooApp, a leaf node that compiled to a static executable it can certainly use its own old version of CoolUtils and not hurt anything around it. There could be major version differences as well, perhaps FooKit is deprecated and supports only swift-4.0, but is kept in the workspace to support easy security fixes. And finally there could be an incremental upgrade problem - perhaps FooKit's constraint came not from itself but from another of its dependencies, CoolerUtils. FooKit would need to wait for upstream to release a new version of CoolerUtils, drop its CoolerUtils dependency altogether, or get kicked out of the workspace so BarKit can upgrade. But we still want to manage FooKit with the rest of the packages we maintain, so soon we'll have a SE-NNNN Meta-Workspaces.

EDIT: I really dislike the date format. I didn't realize that this was an entire year old.

3 Likes