Pre-Pitch: `@package` argument syntax

Hello folks at the Swift community!

A year and a while ago, I started the work of supporting @package(...) import syntax for importing SwiftPM dependencies in Swift scripts as a GSoC project. During the project, I managed to work out a proof-of-concept of this feature with Swift Driver integration. The PoC implementation consists of a small standalone tool that depends on SwiftSyntax, since adding SwiftSyntax dependency to SwiftPM was a big trouble at that time.

With the great work from the Swift Syntax team, we now have a pure-Swift implementation of Swift parser that’s going to be part of the Swift toolchain. This makes depending on SwiftSyntax a lot easier, and I believe it’s time to move on with this feature!

In the former pitch and implementation, the @package attribute takes arbitrary parameters that will be passed to SwiftPM as-is. Now that PackageDescription API is mostly stabilized since the introduction of package registry in SwiftPM 5.6, and SwiftSyntax makes semantic parsing really easy, I'd like to propose the following syntax for @package.

A draft parser implementation (using @_package) is available at apple/swift-syntax#1233. Formal grammar is formatted after the new DocC-based TSPL.


Package attribute arguments consist of two parts: package description which is required, and package product which is optional.

package-attribute-arguments → package-description | package-description , package-product

A package product specifies the product name to import the module from. If there’s no explicit package product, we assume the product to be named after the imported module.

package-product → product : package-product-name

There’re three kinds of package description, matching Package.Dependency.Kind definitions in PackageDescription.

package-description → file-system-package-description

package-description → source-control-package-description

package-description → registry-package-description

File-system package description contains the relative or absolute path of the dependent package.

file-system-package-description → path : package-path

package-path → string-literal

Source-control package description contains the Git URL of the dependent package and package requirement.

source-control-package-description → url : package-url , source-control-package-requirement

package-url → string-literal

Registry package description contains the identifier of the dependent package and package requirement.

registry-package-description → id : package-identifier , registry-package-requirement

package-identifier → string-literal

A package requirement is either a labeled parameter or a version range expression.

source-control-package-requirement → labeled-source-control-package-requirement

source-control-package-requirement → package-version-range

registry-package-requirement → labeled-registry-package-requirement

registry-package-requirement → package-version-range

Package version range contains two semantic version string literals, joined with ..< or ... operator.

package-version-range → semantic-version-string range-operator semantic-version-string

semantic-version-string → string-literal

range-operator → ..< | ...

Labeled package requirement tries to replicate the Package.Dependency static APIs for source-control and registry package dependencies.

labeled-source-control-package-requirement → source-control-package-requirement-label : package-requirement-parameter

labeled-registry-package-requirement → registry-package-requirement-label : package-requirement-parameter

source-control-package-requirement-label → branch | exact | from | revision

registry-package-requirement-label → exact | from

package-requirement-parameter → string-literal

9 Likes

This pitch raises an interesting, more general question: it would encode a SwiftPM-specific concept into the core language. This requires adding a new attribute to the language, but from what I can tell, the attribute would only be used by a separate tool (or driver integration?) that extracts them from the source code—the compiler/frontend itself would never actually use it, correct? (One of your initial implementation PRs added frontend support to extract those attributes into a separate JSON file, but it looks like that's been replaced by the new SwiftSyntax-based tool, if I'm reading it correctly.)

In other words, the attribute is effectively no different than a comment; the same functionality could be expressed this way:

// package(url: "https://github.com/apple/swift-log.git", from: "1.0.0")
import Logging

// package(path: "swift-argument-parser")
import ArgumentParser

Since SPM is not the only build system used for Swift, it feels like a bit of a layering violation to require the frontend to understand SPM-specific concepts, especially if it just ignores/discards them—it still has to be able to parse them. So I'd really like to explore if there's a way we could generalize it so that other build systems can adopt it for their own purposes, and so that if the information has to be provided using syntax written in Swift source in some fashion, that it is actually used by the compiler.

9 Likes

I've also been wondering if this would be a good fit for another compiler plug-in (since we have those now) - a kind of module provider.

The toolchain would ship with one for SwiftPM. Possibly there could be other module providers?

3 Likes

Interestingly this is why I gave -1 in SE-0386 review.

In the new implementation, I tried to not modify the compiler itself. Providing parsing through SwiftSyntax is mostly for developer tools (like IDE) to provide diagnostics, which is one of the remaining pieces of the original PoC.

I believe there’re other ways to achieve this functionality, eg. we may have a new macro type that’s effectively no-op but to provide extra information in a designated format. Or like @Karl suggests we can supply modules through external providers to the compiler as a plug-in (I had considered similar approach, but gave up mostly because it’s challenging to design a module-based global cache system).

Implementing these, however, is beyond my ability as an individual developer. I sincerely welcome more thoughts on how we can do this non-invasively, and support on related infrastructure.

I think we need a way to specify the Swift language mode (or version) in Swift “scripts” so they don’t break as the language evolves.

Maybe SPM should be the shebang script runner so it could preprocess the file?

Maybe even do a multi-line shebang? (maybe less verbose syntax then this)

#!/usr/bin/swift
#!SwiftToolsVersion(5.7)
#!package(url: "https://github.com/apple/swift-log.git", from: "1.0.0")
#!package(url: "https://github.com/apple/swift-argument-parser", from: "1.2.0")
#!product(name: "ArgumentParser", package: "swift-argument-parser")
#!product(name: "Logging", package: "swift-log")

import ArgumentParser
import Logging 
1 Like

The reason I introduced Swift Syntax integration is we want to provide diagnostic for IDEs with existing tools they’re using. Making SwiftPM the default tool is more invasive than we want.

In the PoC a script is passed to SwiftPM only when @package is detected by the driver, so existing scripts are still run by the interpreter. The interpreter is generally faster because it skips the step of evaluating the manifest and building and linking the executable.

I don’t think Swift version is really necessary, but we have better ways to specify it — through a new driver flag, eg.

#!/usr/bin/env swift -require-version=5.8
1 Like

I know not a topic for this, but I think adding that as a parameter would be great. I’ve been burned by the swift version changing using swift as a scripting language for some automatic tasks where I forget to update the scripts.

I think it would be great to see some sort of global cache for packages in this scripting mode, but maybe more in the domain of SPM or other build system.

It would be very cool if Swift Playgrounds adopted this.

Is the idea still to create the Package.swift file automatically?

Unfortunately, this solution isn't portable. If I just google "shebang arguments", the first result I get proves it doesn't work: scripting - Multiple arguments in shebang - Unix & Linux Stack Exchange

2 Likes