Package Manager Workspace

Development Package Manager
1

Hi,

I would love to get some feedback on a draft proposal for introducing a feature called "workspace".

https://github.com/aciidb0mb3r/swift-evolution/blob/workspace/proposals/NNNN-package-manager-workspaces.md

Package Manager Workspace

Introduction

This proposal introduces a new feature called "workspace" to help Swift package
authors to develop packages that depend on each other.

Motivation

Swift package authors usually want to develop their set of packages in tandem.
This is currently possible using the swift package edit feature. However,
that feature is suitable for one-off edits rather than continued development.
Moreover, edited packages are not persistent and thus can't be shared with the
development team.

Package authors also want to know the impact on downstream dependencies before
committing a change. There is no obvious way to do this right now but there are
some options like creating a pre-release tag or editing the package in the
downstream dependency. Both of these are suboptimal solutions.

Proposed solution

We propose a new feature in package manager called "workspace". A SwiftPM
workspace will be driven by a new manifest file called Workspace.swift. The
file will contain a list of packages and each package will act as a root
package. The dependency resolver will combine dependency requirements from each
root package before resolving them. This means that it is possible that
a certain root package is overly constraining a dependency requirement.

Detailed design

  • Initially, we will introduce the following APIs for Workspace.swift file:
/// Defines a package workspace.
final class Workspace {

    /// Represents a package inside a workspace.
    final class Package {

        /// The URL of the package.
        ///
        /// This will be used to clone the package at `path` if it doesn't exist.
        let url: String?

        /// The local path of the package.
        let path: String

        /// Any custom user data associated with the package.
        let userInfo: JSON?

        private init(
            url: String?,
            path: String,
            userInfo: JSON?
        ) {
            self.url = url
            self.path = path
            self.userInfo = userInfo
        }

        static func package(
            url: String? = nil,
            path: String,
            userInfo: JSON? = nil
        ) -> Package {
            return Package(url: url, path: path, userInfo: userInfo)
        }
    }

    /// The packages in the workspace.
    var packages: [Package]

    init(packages: [Package]) {
        self.packages = packages
    }
}

Example:

// swift-tools-version:4.2
import PackageDescription

let workspace = Workspace(packages: [

    .package(
        url: "https://github.com/apple/example-package-dealer",
        path: "../dealer",
        userInfo: .init(["branch": "master"])),

    .package(
        url: "https://github.com/apple/example-package-fisheryates",
        path: "../fisheryates"),

    .package(
        url: "https://github.com/apple/example-package-deckofplayingcards",
        path: "../deckofplayingcards"),

    .package(
        path: "../playingcard"
        userInfo: .init(["fork": "https://github.com/aciidb0mb3r/example-package-playingcard"])),
])

  • The url property is optional and will be used to clone the root package if it
    does not exist at path.

  • The userInfo property allows associating arbitary data with a package. The
    package manager will not read its content and will dump it as-is when JSON
    representation of the workspace manifest is asked.

  • If some root package is also a dependency in the overall package graph, the
    dependency requirement will be ignored and the root package will be used
    instead.

  • The package manager will build all root packages in the same build directory
    which is controlled by the --build-path option.

  • The package manager will write a file called Workspace.resolved
    to record the dependency resolution result. The behaviour of this file will be
    completely similar to the existing Package.resolved file. Generally, this
    means, if Workspace.resolved is present, the package manager will
    prefer the recorded versions.

  • It will not be allowed to have both Package.swift and Workspace.swift
    in the same directory. We may lift this restriction in future by adding an
    option to prefer one over the other.

  • Package authors can check-in this file in a repository and share the workspace
    with a development team. Note that this file can be used to create different
    workspaces for automated testing of downstream dependencies.

Impact on existing packages

None.

Alternatives considered

None.

9 Likes
2

I think this idea would make for a nice addition.

Q: why move away from using Swift to define the workspace?

3

I think we may ultimately want that but right now it comes with a lot of restriction and is slow to parse/load/execute. People have different workflow needs and they can easily customize by reading or manipulating this JSON file. OTOH, we could add a way to dump JSON if we use a Swift file and then tools can work with that. But I think that'll be slow and painful in practice.

1 Like
4

My immediate response was to ask the same question.

That's true, but isn't it true for Package.swift too? Maybe I'm being too dogmatic but it feels inconsistent to push Swift as the format for one and not the other.

4 Likes
5

For Xcode users this might be a bit of overloading on the term "workspace". Or is the idea to have a SPM Workspace be analogous to a Xcode Workspace?

1 Like
6

I think we can easily expand this feature in future to use a Swift file and the JSON option can remain for simpler use cases. This proposal aims to solve problems that community is facing right now. Using Swift file is more consistent and powerful but it comes with a lot of complexity.

2 Likes
7

We shouldn't need to worry about what term Xcode uses as SwiftPM is a separate software (also, Xcode is not part of Swift OSS project). That said, I don't strongly feel what we call this feature and I am open to naming suggestions.

1 Like
8

The overall idea sounds great, but the inconsistency of using JSON instead of the existing Swift format strikes me as a something that’ll cause dramas down the road.

I’d want to write the package/workspace descriptions in the same format.

7 Likes
9

Thanks for the feedback Tony! I am not sure what you mean by drama. I updated the proposal to explain why I think we should use JSON for now (and then move to a Swift based format in future).

1 Like
10

Sorry, yes, I’ll clarify: once the JSON format is deployed, rolling it back could take a long time. It also would likely encourage people to ask for the same for standard package manifests.

I know it’s not the easier technical solution, but I feel that using a consistent format for both package and workspace manifests is a better experience for developers/users of SPM.

5 Likes
11

I don't think supporting a subset of options will be a technical burden so we might not necessarily need to roll it back instantly. However, if it turns out that no one wants to use the JSON file because everyone has moved to the Swift based format, we can easily deprecate and remove it.

I completely agree that using a Swift based format will provide a consistent and better experience in future. I think the point I am trying to make is that we don't lose anything by having a JSON based format.

12

It does feel like it adds churn, if we think it should ultimately be Swift to be consistent. I tend to agree with Tony here...

3 Likes
13

Moving to Swift based file will also depend on how this feature evolves over time and with what complexity. I don't think it is always good to implement something which provides way less flexibility just to be consistent with other parts.

1 Like
14

I agree with @Aciid here, JSON is so easy to parse, doesn't need any additional tooling and Swift workspace definitions can be added later. I find the fact that pure package manifests can't be declared in JSON or some other easily parsed language very inconvenient. If consistency is desired, adding a possibility to have JSON package manifest would work much better.

1 Like
15

The term "workspace" is already widely used when working with other package managers, where it has the same meaning, check out Workspaces | Yarn

1 Like
16

I get that, but my sense (from other discussions I've read too) is that there seems to be a bit of underlying tension between Swift and non-Swift solutions - both here and even for the package format itself.

I think that tension needs resolving.

If you go for JSON here for pragmatic reasons, it somewhat undercuts the arguments for sticking with Swift for the package. I'm hoping that there will be plans sooner rather than later to expand the scope of the package manifest to support settings and more sophisticated build operations. They are also problems that the community is facing right now, and an argument could be made for a solution to them that just involves JSON too, for similar reasons (it's simpler, we can get it up and running faster, etc).

I'm not arguing for one or the other (I can see pros & cons both ways), I'm just arguing for consistency and the sense that there's a long term plan that's being followed, rather than a series of short-term decisions which might leave us with a bit of an incoherent picture over time.

7 Likes
17

I agree there are problems that needs to be addressed in other parts of the package manager. This feature is orthogonal to other outstanding issues and missing features. This proposal does not aim to solve or indirectly influence other parts of the package manager.

Right. We definitely don't want to put ourselves in a position that we cannot recover from. I've already mentioned how this feature can evolve overtime and move away from JSON in a clean way.

1 Like
18

I had more discussions about this proposal and its clear that Swift is much more preferable and consistent. I completely dropped JSON based manifest from the proposal and introduced a Swift based manifest.

PS: I am still thinking if we actually need the userInfo property in the Swift based manifest.

3 Likes
19

I have difficulties to understand what workspace is meaning to work.
At first I though it was about defining custom package dependency overload (of Package.swift) while developing a package. But at the end you say:

It will not be allowed to have both Package.swift and Workspace.swift
in the same directory. We may lift this restriction in future by adding an
option to prefer one over the other.

So I'm confused. What's the role of Workspace? How does it integrate with existing SPM and especially Package.swift if we can't have both of them?

20

Same for me. I don't get the objective of this proposal, for what would be used? It seems it has the same objective as the Local Dependencies proposal.