Can we prefer local package and fallback to url package

Prefer local package and fallback to url package

Introduction

Add a new method to PackageDescription.Package.Dependency to create a package.

Motivation

Currently we use some logic like the following to toggle between local package and remote packages.

if ProcessInfo.processInfo.environment["USE_LOCAL"] == nil {
    package.dependencies += [
        .package(url: "git@github.com:apple/swift-docc.git", .branch("main")),
    ]
} else {
  	// For CI or local use
    package.dependencies += [
        .package(path: "../swift-docc"),
    ]
} 

But if we depend a lot of repo and we only have some of them locally, we have to either clone them all locally or use the remote version.

if ProcessInfo.processInfo.environment["USE_LOCAL"] == nil {
    package.dependencies += [
        .package(url: "git@github.com:apple/swift-docc1.git", .branch("main")),
	      .package(url: "git@github.com:apple/swift-docc2.git", .branch("main")),
        .package(url: "git@github.com:apple/swift-docc3.git", .branch("main")),
        .package(url: "git@github.com:apple/swift-docc4.git", .branch("main")),
        .package(url: "git@github.com:apple/swift-docc5.git", .branch("main")),
    ]
} else {
  	// We only have swift-docc2 and swift-docc5 locally
    package.dependencies += [
        .package(path: "../swift-docc1"),
      	.package(path: "../swift-docc2"),
        .package(path: "../swift-docc3"),
        .package(path: "../swift-docc4"),
        .package(path: "../swift-docc5"),
    ]
} 

I proprosal to add a new method to PackageDescription.Package.Dependency so that we can declare a local first package and if the local package does not exist we can use to remote one.

Proposed solution

extension Package.Dependency {
    static func package(preferLocal: Bool = true, localPath: String, url: String, _ requirement: Package.Dependency.Requirement) -> Package.Dependency {
        if preferLocal, FileManager.default.fileExists(atPath: localPath) {
            return .package(path: localPath)
        } else {
            return .package(url: url, requirement)
        }
    }
}

Currently due to the sandbox limitation, FileManager.default.fileExists(atPath:) will always return false. And we need to add --disable-sandbox to workaround.

My proposal is to add a method, so that we can get a bool result for whether a location has a valid package

or not. Something like this :point_down:

func successResolvePackage(at path: String) -> Bool

Alternatives considered

Use --disable-sandbox to workaround.

https://github.com/apple/swift-docc/blob/0d61dc5550c9bdb04fd1579144679e7d3287f640/Package.swift#L116-L141

2 Likes

Inspired by this PR, found another workaround. Enable library evolution for SwiftDocC library via a conditional by franklinsch · Pull Request #123 · apple/swift-docc · GitHub

We can use #filePath and deletingLastPathComponent API to workaround with the sandbox limitation.

Note: ".." will not work, use deletingLastPathComponent instead

extension Package.Dependency {
    static func package(
        preferLocal: Bool = true,
        name: String,
        location: String? = nil,
        url: String,
        _ requirement: Package.Dependency.Requirement) -> Package.Dependency {
        let manifestLocation = URL(fileURLWithPath: #filePath)
        let location = location ?? name
        let packageLocation = manifestLocation
            .deletingLastPathComponent()
            .deletingLastPathComponent()
            .appendingPathComponent(location)
        if preferLocal, FileManager.default.fileExists(atPath: packageLocation.path) {
            return .package(path: packageLocation.path)
        } else {
            return .package(url: url, requirement)
        }
    }
}

Using the above method, we could check the existence of some folder in the parent directory of the Package without adding --disable-sandbox flag. Is this a SPM sandbox bug or a known use case? cc @tomerd

#filePath will report wherever the manifest was loaded from. This is not necessarily the same location you are looking at as a developer.

In your particular use case, once someone else depends on your package, there is a high risk that your manifest, which is now in a temporary build directory, will detect a neighbouring dependency. If it does, it will report back a relative path from behind a version number, which is illegal and will invalidate the graph.

Do not be tempted to use #filePath to read from other files in the repository either. A package in a registry will most likely have the manifest fetched and loaded in isolation first to resolve versions before even asking for the rest of the source.

These are just two examples. But it is not a supported feature, and SwiftPM reserves the right to change the implementation details of how, when and where it loads the manifest. So even what seems to work now might suddenly stop working at any time.

A better strategy for your use case would be to switch on an environment variable instead. Reading from the environment is a supported feature, works reliably behind any kind of dependency, and is widely used in the Swift project itself. (There is a caveat that some IDEs caches won’t detect staleness when the environment changes and need to be manually purged. But SwiftPM proper handles it correctly.)